Automating government tender discovery: the TrackTender approach

Bidding teams lose deals not because they write bad proposals, but because they find tenders late — or miss a corrigendum entirely. Here is how we turned dozens of procurement portals into one queryable, alert-driven system.

Ketul Shah — Anekant AI

The problem: procurement lives in too many places

If your company bids on government work in India, someone on your team starts every morning the same way: opening a stack of e-procurement portals — central, state, and department-level — and running the same searches they ran yesterday. Each portal has its own login quirks, its own search form, its own idea of what a category is called. The tender you care about might be filed under "supply of IT hardware" on one portal and "computer peripherals" on another.

And browsing is only half the job. Tenders change after publication. A corrigendum can move the submission deadline, revise the estimated value, or alter eligibility criteria — and it is published as yet another document on the same portal, easy to scroll past. Miss the notice that a deadline moved up, and a bid your team spent two weeks preparing simply never gets submitted. That is not a hypothetical failure mode; it is the one bidding teams describe first when you ask what hurts.

The manual approach does not scale past a handful of portals, and it fails silently. Nobody knows what they did not see.

Why this is a data engineering problem, not a browsing problem

The instinct is to treat this as a browsing chore to delegate — hire someone to check the portals, or record a browser macro. Both break for the same reason: the portals are not one system. They are dozens of independently built systems that happen to serve similar content.

• Structure varies wildly. Some portals render clean HTML tables; others hide listings behind POST-only search forms, session tokens, and CAPTCHAs on deep pages. Pagination might be a query parameter, a form resubmission, or a JavaScript click handler.
• The real content is in PDFs. The listing page gives you a title and a date. The scope of work, eligibility, EMD amount, and pre-bid meeting details live inside attached tender documents that need downloading and parsing.
• Records mutate. A tender is not a row you fetch once. Corrigenda amend it after publication, which means yesterday's copy of the data can be wrong today. Any system that only collects and never re-checks will confidently report stale deadlines.

Once you frame it that way, the requirements become familiar data engineering requirements: heterogeneous ingestion, normalization into a common schema, change detection over time, and event-driven notification. That framing is what shaped TrackTender.

What we built

TrackTender is a tender discovery and tracking system with four layers, each deliberately boring.

1. Scheduled collectors, one per portal. Each procurement portal gets its own small Python collector that knows that portal's quirks — how to search, how to paginate, how to fetch attachments. Collectors run on schedules tuned to how often each portal actually updates, and every run is logged so a silent failure on one portal surfaces as a monitoring alert, not a gap nobody notices.

2. Normalization into one schema. Whatever shape the source data arrives in, it is mapped into a single relational tender record: portal-scoped tender ID, issuing authority, category, value band, and the dates that matter — published, pre-bid, submission, opening. Search and alerting only ever run against this normalized store, so adding a new portal never touches the layers above it.

class Tender(Base):
    portal        = Column(String)   # source system
    tender_ref    = Column(String)   # portal's own ID
    authority     = Column(String)   # issuing department
    category      = Column(String)   # normalized taxonomy
    value_band    = Column(String)   # e.g. "10L-1Cr"
    published_at  = Column(Date)
    submission_by = Column(DateTime) # the date that matters
    opening_at    = Column(DateTime)
    content_hash  = Column(String)   # for change detection

    __table_args__ = (
        UniqueConstraint("portal", "tender_ref"),
    )

3. Change detection for corrigenda. Every collector run recomputes a content hash over the fields that matter for a live tender. If the hash for an existing record changes, we diff old against new and record exactly what moved. A shifted submission deadline is not a quiet row update — it is an explicit event with its own severity.

def detect_changes(existing, fresh):
    if existing.content_hash == fresh.content_hash:
        return []

    events = diff_fields(existing, fresh)

    # a moved deadline is a first-class event
    if fresh.submission_by != existing.submission_by:
        events.append(DeadlineChanged(
            tender=existing,
            old=existing.submission_by,
            new=fresh.submission_by,
            severity="critical",
        ))
    return events

4. Subscriptions and alerts. Users subscribe to keywords, categories, authorities, or value bands. New tenders matching a subscription trigger a notification; so does any change event on a tender the team is tracking. Deadline alerts escalate as the date approaches, so a submission due in 48 hours does not rely on someone remembering to check a dashboard.

Lessons for developers building something similar

Politeness is architecture, not etiquette. Rate limits, respectful scheduling, caching, and conditional requests are what keep a collector running for months without getting blocked — and they are also just good citizenship toward public infrastructure. Build the throttle into the collector framework so no individual scraper can misbehave.

Prefer resilient selectors over precise ones. A selector pinned to div:nth-child(3) > table > tr breaks the first time a portal adds a banner. Anchoring on semantic signals — a header cell labelled "Closing Date", a link whose text contains the tender reference — survives cosmetic redesigns. When a portal does change structurally, you want the collector to fail loudly and fast, not to ingest garbage quietly.

Deduplicate across portals, not just within them. The same tender often appears on both a central aggregator and a department portal, with slightly different titles. A composite key of authority plus normalized reference number, with fuzzy title matching as a fallback, keeps one real-world tender from becoming three noisy alerts.

Treat "the deadline moved" as a first-class event. This was the single most valuable design decision in the project. Most scraping systems model the world as a snapshot; the users' actual risk lives in the transitions. Modelling changes as events — with types, severities, and their own delivery rules — is what turns a scraper into a monitoring system.

Where else this pattern applies

Strip away the procurement specifics and TrackTender is a general shape: watch many inconsistent sources, normalize what they publish, and never miss a change. The same architecture — collectors, one schema, change events, subscriptions — carries directly to regulatory and compliance updates across government gazettes, competitor price lists and catalogue changes, market notices and exchange circulars, and grant or RFP announcements across funding bodies.

If your team has a "someone checks the websites every morning" ritual anywhere in the business, that ritual is a pipeline waiting to be built. It will check more sources than a person can, it will not get bored, and it will tell you the moment something changes rather than the next morning.

You can read more about how we approach this kind of work on our data pipelines and scraping services page, or see TrackTender alongside our other builds in the portfolio.