Freelance Radar: a Go monitoring and scoring pipeline

Context

On freelance marketplaces the first suitable bid wins, not the best one. New listings collect their first replies within minutes, and on one of the two marketplaces I track, more than half of the relevant listings get no bids at all. Watching several feeds by hand does not scale: you either miss work or spend the day refreshing tabs. Freelance Radar is my answer - a Go pipeline that monitors two freelance marketplaces, scores every listing and delivers the good ones to Telegram. I built it for myself and run it daily.

The problem

Each marketplace has its own format. One publishes an RSS feed but hides the full description and the budget on the listing page; the other embeds ready JSON inside its HTML and forbids pagination in robots.txt. Budgets arrive in different currencies and shapes: fixed price, hourly, “negotiable”. The same job is often posted on both marketplaces at once. On top of that comes noise: fake listings, homework, “pay with a review” - jobs that should get zero time and zero LLM calls.

The solution

A four-step pipeline: collect, enrich, score, deliver to Telegram. The steps never call each other - they are decoupled through PostgreSQL tables, and “queued for the next step” is simply the absence of a related row, so a crash in any step does not take the others down. Collection polls sources within an active window: FL.ru every 2 minutes, Kwork every 3; outside the window the intervals stretch. The two-minute step on FL.ru is not a speed choice: DDoS-Guard lets roughly one request per minute or so through from a single address, and polling faster would simply hit the shield. Scoring is two-tier: free rules run first - a catalog of service categories, freshness, budget, competitor count, stop words; in the grey zone a language model steps in, shifts the score within a fixed cap and explains why. A listing that clears the threshold arrives as a Telegram card: score breakdown, budget in the original and the base currency, red flags, and two buttons - “Interesting” and “Pass”.

collect  -> enrich  -> score -> deliver
markets     details    rules    Telegram
2-3min      budgets    + LLM    2-3 min

The result

Matching listings land in Telegram 2-3 minutes after publication, with an explanation of why the listing scored what it did. Button reactions accumulate in the database and have already driven the first weight calibration: new stop words, a wider grey zone. The collected base of 1047 listings became the dataset for a demand analysis that shaped my service catalog and this site.

What was hard

• Cross-marketplace deduplication. Duplicates are caught by a content fingerprint: SHA-256 of the normalized text, checked against everything delivered within a time window. A duplicate is flagged, not deleted - silently losing a job is worse than an extra message. A failed check is treated the same way: the pipeline fails open toward delivery.
• Multi-currency budgets. The exchange rate is pinned the moment a listing enters the system and stored with it; rate fetching is a cascade with fallbacks. Otherwise re-scoring a week later would produce a different number with no visible reason.
• Two-tier scoring. Rules are cheap but cannot tell “a bot that messages customers” from “a bot that fakes engagement”. The model is called only where the rules are unsure. Any model failure - timeout, rate limit, invalid response - means the rule score ships unmodified: the listing must reach the chat. Model calls are capped per day.
• Migration rollback guards. A schema rollback refuses to run if it would silently distort data already in the database. A single step is allowed only from the latest version; a multi-step rollback only with an explicit target.
• Per-source circuit breaker. Five consecutive errors from a source pause it for an hour and raise a Telegram alert. Partial results survive a failed run: one bad request does not erase what was already collected.
• Zero is not “unknown”. One marketplace reports the number of competing bids, the other does not. The rule “few bids is a plus, many is a minus” distinguishes zero from missing data and never penalizes a listing because the marketplace stays silent.