In brief: TENDER is a system that walks 29 international platforms every day, collects every tender published by UN agencies and development banks, and shows the expert only the ones actually worth bidding for. The judgement is made by a language model deployed on the client’s machine: neither the CV nor the terms of reference ever leave for an outside service.
Client: independent expert consultant · Category: scraping · local LLM · Telegram bot
What matters most in this project?
- 300+ new tenders a day across UN platforms; only a handful reach the expert.
- 100% selection accuracy: nothing irrelevant in the output, nothing to sift through.
- The AI runs locally. A CV is personal data and some terms of reference are covered by client confidentiality, so the model sits on their machine, not in the cloud.
- Zero token cost: spending does not grow with the number of tenders processed.
- 29 sources behind one interface: UN agencies, development banks, EU and OSCE portals.
- Control from Telegram: opportunities arrive in chat, get rejected there, and the admin panel opens from the same place without a password.
What is the project built with?
- Scraping:
- Python 3.12, httpx, BeautifulSoup, lxml. For JavaScript sites (the Asian and African banks) Playwright through Bright Data Scraping Browser: a real Chrome rendered remotely, with no local browser.
- Anti-bot:
- Bright Data Web Unlocker for Cloudflare and JS, Scraping Browser for full rendering.
- AI:
- a local model, Ollama plus Qwen 2.5 with an OpenAI-compatible API, deployed on the client’s hardware. Prompts are in Russian and editable from the admin panel.
- Backend and admin:
- Flask 3 with gunicorn, Jinja2 templates, custom CSS with two themes and a mobile version.
- Bot:
- aiogram 3 on long polling, with no open ports.
- Database:
- SQLite, isolated, no shared MySQL.
- Export:
- openpyxl for Excel, reportlab for PDF with Cyrillic.
- Infrastructure:
- nginx with its own block and basic auth, systemd running four services and timers, a separate server user.
- Integrations:
- Bright Data balance API, National Bank of Kazakhstan exchange rate.
No external AI service is used at all. The model can be swapped without rewriting the pipeline: the interface is OpenAI-compatible.
How did it start?
UN tenders are competitive procurements run by UN agencies and development banks: consulting, expert reviews, research. The notices are scattered across dozens of sites, among them UNGM, UNICEF, UNDP and the World Bank. More than three hundred new ones appear every day, and going through them by hand takes hours daily.
Volume is not the only problem. A tender title almost never describes what is inside: a project may be called “Public financial management” while the actual role wants an ecologist or a local consultant with mandatory command of the country’s language. You cannot filter that by the title, you have to read the full description.
Handing that work to a cloud AI was not an option. The model’s context takes the expert’s full CV, which is personal data, and some terms of reference are covered by client confidentiality. So the model was deployed on their own machine.
What was built?
🧠 Why does the LLM run on the client’s hardware?
- Ollama and Qwen 2.5 on the client’s machine: the match decision is made locally
- CVs and terms of reference never go to external APIs, neither the cloud nor someone else’s logs
- Two deployment options, the client’s own hardware or a rented server, with the same privacy perimeter
- Zero token cost: spending does not grow with the number of tenders processed
- An OpenAI-compatible interface: the model can be swapped without rewriting the pipeline
- Prompts are editable from the admin panel and the full model dialogue is stored for every opportunity
🤖 How do the scrapers and collection work?
- 29 sources behind a single Connector → Vacancy interface: UN agencies (UNICEF, UN Women, UNFPA, IOM, UNGM, UNDP), banks (World, Asian, Islamic, African, EBRD, EDB, EFSD), EU and OSCE (TED, OSCE, ILO)
- Five protocol types under one abstraction: server-rendered HTML, REST JSON API, Oracle Recruiting Cloud (one engine for three agencies), POST with an HTML fragment, sitemap XML
- Real-browser rendering for JS sites where the listing is assembled by a client-side script
- An AI recovery controller: if a source changes its markup and the scraper breaks, the model reconstructs the card fields, grounded in the HTML and capped against runaway calls
- One failing source never brings down a run; there is throttling with pauses and per-source deduplication
⚙️ How does the processing pipeline work?
- Normalisation: dates from six incompatible formats to ISO, countries from NUTS codes and free-form strings to a name and region using a 249-country ISO 3166-1 reference
- Country resolution: strict text heuristics with an AI fallback when nothing matches
- Cross-source deduplication: the same tender sits on several portals, the system merges duplicates and keeps the fullest record, enriching it from the rest
- A two-level prefilter: STOP words, CORE words, territorial filter, language, national-consultant flag, deadline. Every rule is editable in the panel and the log tags which ones fired
- Expired notices are dropped at two points: in the prefilter and again after details load, because banks only reveal the deadline inside
🌍 How does the territorial filter work?
- A 249-country reference: ISO 3166-1, Russian and English names, codes, normalisation
- An editable allow-list of countries with search across all 249
- Applied twice: in the prefilter and again after the model’s verdict
🧠 How does the AI match tenders?
- The model receives the expert’s complete CV: 69 projects, publications, education, languages, around 34,000 characters, not a summary
- Judgement by the substance of the role rather than the title: the model reads the full description and rejects mismatches
- A structured verdict: yes or no, a short rule-based reason, a long descriptive reason, a summary in Russian and English, the country and a terms-of-reference flag
- Calibration proven against reference cases, with a full log kept for every opportunity
- Prompts and the profile are editable from the panel, with safeguards against breakage
🖥 What does the 13-page admin panel do?
- Dashboard: real-time server health, a graphical funnel of every phase, the status of each scraper
- Finance: spend per run, a monthly forecast by median, the cost of a run and of each opportunity found, proxy balance with a top-up button
- Selected opportunities with a detail card: full description, attachments, delivery log, manual rejection with a reason and a STOP word
- Accepted, rejected and a log of all opportunities with search and filtering by rule tags
- A card for each of the 29 sources: what it collects, method, anti-bot, schedule
- STOP and SIGNAL editors, a territory tab, logs, the algorithm with its real prompts, a prompt editor, recipients, settings
- Export of lists to CSV, Excel, JSON and PDF, respecting the active filter
- Admin user management and login by link from Telegram
- Two themes, responsive layout, a card-based mobile version
🤖 How does the Telegram bot work as a control panel?
- A menu with permanent buttons: all opportunities, sources, statistics, help
- Automatic delivery of every new matching opportunity to active recipients
- A “Not a fit” button rejects an opportunity straight from the chat, in sync with the panel
- Attachments and terms of reference arrive as documents
- Extended statistics: funnel, database, balances, efficiency
- Password-free entry to the admin panel through a personal link
🚀 How are automation and production set up?
- A daily pipeline on a systemd timer: collect, AI, deliver, catching up on missed runs
- A report scheduler: an hourly timer checks who is due, with interim reports at a chosen hour and a weekly one on Mondays, in each recipient’s time zone
- Manual start with a “Run now” button
- Four production services, isolated from other projects on the server
💳 How is financial tracking handled?
- Spend per run: proxy and browser traffic, with no token bill
- Efficiency: the cost of a run and of each opportunity found, plus a monthly forecast
- Live proxy balance over the API with a top-up button
What is the result?
| Metric | Value |
|---|---|
| New tenders a day across the platforms | 300+ |
| Sources monitored | 29 |
| Selection accuracy | 100% |
| Countries in the filter reference | 249 |
| CV projects fed into the model | 69 |
| Context per decision | ~34,000 characters |
| Admin panel pages | 13 |
| Production services | 4 |
| AI token bill | none, the model is ours |
Where to learn more about this case?
The full story of why tender titles lie and how we taught an AI to read UN tenders properly: read the article on LinkedIn.