Semantic Search over GCN Circulars: Text Cleaning, Model Selection and Dimensionality

Introduction

GCN Circulars [4] and Astronomer's Telegrams [5] are the primary channel through which observers report follow-up of transient events. NovaTrace holds 13,247 of them, and keyword search over them retrieves only documents that share vocabulary with the query. Dense embeddings permit retrieval by meaning: a query for "short burst with a kilonova candidate" should return the circulars that describe one, regardless of the terms their authors chose. The work is motivated by three applications:

• Search. The NovaTrace search interface is being redesigned, possibly as an assistant-style surface, and embeddings form its retrieval layer.
• Linking. 94.8% of circulars are linked to a canonical event. Most of the 686 unlinked circulars identify their event only in prose, which embeddings can match.
• Claim grouping. Clustering all statements about a single quantity, such as every redshift claim for an event, prior to testing them for consistency.

Text preparation

We embed the subject and body of each circular, with tables replaced by a one-line summary and author lists removed. Without this step, the largest circular (GCN 39259, 270,801 characters) consists almost entirely of a single 2,230-row MASTER photometry table, and its vector would represent numerical content rather than the substance of the report. After cleaning, it is 2,183 characters long. The rules are deterministic and involve no language model:

• Tables. A line containing two or more pipes, or three or more space-separated columns that are predominantly numeric, is classified as a table line. Three or more consecutive table lines form a table, which is replaced by a summary such as [Table: 253 rows × 8 columns (Tmid-T0, Date Time, Site, …)].
• Author lists. The first paragraph is removed when it consists mostly of names, or when it ends with "reports on behalf of … :". Authors are stored separately.
• Headers. GCN and ATel header lines are removed, except for the subject and the ATel subject tags.

The principal correctness criterion is that no prose is removed. A dry-run report lists every removed line that resembles a sentence; under the final rules it lists none. The report exposed two defects during development: a sentence containing double spaces that was misclassified as a table header (GCN 39922), and 218 short MASTER tables that were missed because a blank line separates their header from their rows.

The same table detector now also serves the extraction pipeline, replacing an earlier parser. It identifies tables in 1,920 circulars, compared with 1,006 for its predecessor, and it corrects a long-standing error in which tables with a second header line of units stored every value under the unit ("deg") rather than the column name ("RA").

Cleaning also resolves the concern over context length. Before cleaning, 140 circulars exceed a 2,048-token window and 9 exceed 8,192 tokens. After cleaning, 1 of 13,247 exceeds 2,048 tokens and none exceeds 8,192.

Model selection

We considered EmbeddingGemma, EmbeddingGemma 2, BGE-M3 [3], the Voyage hosted API and the Qwen3-Embedding family [1]. EmbeddingGemma was the initial candidate but was set aside because of its 2,048-token context window. We selected Qwen3-Embedding for three reasons: it is served by Featherless, the inference provider already used for extraction; its weights are open, which avoids dependence on a single vendor's vector space; and its vectors support Matryoshka-style truncation [2]. A vector may be truncated to its first N components and re-normalised while remaining useful, so a single embedding run can be evaluated at every candidate stored size.

We began with the smallest model, 0.6B, at $0.01 per million tokens. Pricing the larger models showed that 8B is billed at the same rate (Table 2).

Circular text averages 2.91 characters per token, against an assumed 3.5. The corpus is therefore approximately 5.9M tokens, and a single live circular approximately 410 tokens, or $0.000004. At this price, cost ceases to be the binding constraint. The binding constraint is that the API token is shared with live ingestion.

Bulk embedding on a shared API token

A single Featherless token serves live circular extraction, event summaries and cross-matching, and a rejected live extraction call results in a placeholder record. The bulk job is therefore designed to yield to live traffic (Figure 1):

• Serial requests. One request at a time, in batches of 16 circulars.
• A duty cycle of at most 50%. After each request the job waits at least as long as the request took, so that a live call usually finds the slot free.
• Exponential backoff on contention. Once the client's own retries (approximately 15 seconds) are exhausted, the job waits 30 s, 60 s, 120 s and so on, up to 10 minutes, with jitter.
• A content-addressed cache. Each vector is stored at full size on arrival, keyed by a hash of its text, so an interrupted run resumes without repeating work and no text is embedded twice.

Under this policy the full corpus took 51 to 56 minutes per model, with no contention backoffs.

Validation of inputs and outputs

Two of the 13,247 vectors returned by the 8B model consisted entirely of NaN values, all 4,096 of them, for two short "duplicate submission" notices (GCN 40414 and 40254). The run did not detect this, because both API calls succeeded. The database did: its vector type rejects NaN, and the load halted after 1,100 rows. Resubmitting the same two texts returned valid vectors, which indicates a transient fault in the hosted model rather than a property of the input. This was the first sign of the fault examined in a later section.

Inputs required validation as well. Five older circulars (GCN 40549 to 40553) had retained their complete email record, serialised as JSON, in the raw-text field, so their vectors were computed from escaped JSON rather than from the circular. The text builder now unwraps such records first, and the five circulars were re-embedded. A per-row assertion that the cleaned text begins with the subject would have detected the problem immediately.

Evaluation

The evaluation uses existing ground truth: 11,844 circulars linked to 1,621 events. Each circular is used as a query to retrieve the other circulars of its own event from all 13,247. We report mean reciprocal rank (MRR) and the proportion of queries for which the top result, or any of the top ten, is correct.

The central finding is that embeddings group circulars by report type rather than by the event they describe. In 294 of 400 sampled queries, the nearest circular was a report of the same type about a different event, at a cosine similarity of 0.93 to 0.95:

• "GRB 250108A: Swift ToO observations" → "GRB 250106A: Swift ToO observations"
• "EP250321a: refined analysis of the EP-WXT and EP-FXT observations" → the same title for EP250315a
• "Fermi GRB 260526A: Global MASTER-Net observations report" → the same title for GRB 260926A

The evaluation therefore measures event identity, which is precisely where dense vectors are weakest: event names and identifiers differ by one or two characters, a distinction that a dense vector does not preserve. It is consequently a weak measure of the semantic search we intend to build, but a reliable measure of relative differences between models and sizes on the same corpus, which is how we use it. Two conclusions follow. First, event identity should be resolved by exact matching, so search must combine exact matching on names and identifiers with vector similarity. Second, grouping by report type is valuable in its own right: a query such as "other bursts where Swift/UVOT reported only upper limits" is well served by these vectors and poorly served by keyword search.

Comparison of 8B and 0.6B

We embedded all 13,247 circulars with the 8B model (5.98M tokens, approximately $0.06, 56 minutes) and repeated the evaluation on the identical corpus. At equal storage, 8B improves MRR by approximately 12% (Table 3).

The aggregate metrics understate the difference. On hand-written queries, the 0.6B model performed well on the type of observation a circular reports and poorly on interpretive physics, and it is on the latter that 8B improves most (Table 4). The larger model nonetheless still groups by report type before event, so exact matching remains necessary.

Stored dimensionality

Storage cost depends on the number of dimensions retained, not on the model that produced them, so the choice of model and the choice of stored size are independent decisions. 8B stored at 1,024 dimensions occupies exactly the space of 0.6B and retains 99.7% of its full-size score; even at 256 dimensions it exceeds the 0.6B model at 1,024 (Figure 2). Storing all 4,096 dimensions would quadruple the table, making it larger than any other table in the database, for a gain of 0.3%.

Effect of corpus size

A remaining concern was that 13,247 circulars might be too few to reveal the value of the additional 3,072 dimensions, and that the gap might widen once the archive brought the corpus to approximately 45,000. We first tested the trend by drawing random samples of the 8B corpus at three smaller sizes, three samples at each, and scoring 1,024 dimensions against 4,096 on each. If additional data increased the value of the extra dimensions, the ratio would decline as the corpus grows. Instead, it remains near 100% at every size and its variance decreases. Once the archive had been embedded, we repeated the measurement on all 44,615 circulars: 1,024 dimensions retain 99.4% of the 4,096-dimension score (Figure 3), confirming that the gap does not grow with corpus size.

Scope of the evaluation

Three kinds of test preceded the question benchmark described below, and only the first has scored ground truth:

1. Circular-to-circular retrieval. The query is a whole circular and the correct answers are all other circulars of the same event, scored by top-1 and top-10 accuracy, recall at 10 and MRR. It is automatic and large (approximately 11,800 queries), and therefore a fair basis for comparing models and stored sizes, but it measures event identity between long documents rather than the short interpretive questions that users type.
2. Hand-written queries. Six queries judged by reading the results (Table 4). These are closest to real use, but have no labelled answers and are too few to score.
3. An out-of-scope distance test. Ten in-domain questions fell at an L2 distance of 0.58 to 0.85 from their nearest circular, ten off-domain questions at 1.01 to 1.16, and the ambiguous query "black hole" between them at 0.92. This establishes a cut-off for questions the corpus cannot answer; it does not assess whether results are correct.

A first question benchmark

To evaluate retrieval for questions rather than whole circulars, we built a benchmark of 18 questions written by Gemini, each labelled with the circular or circulars that answer it, and scored it over all 44,615 cached circulars. The labels themselves required correction: Gemini had fused its footnote markers onto the circular numbers, so that "GCN 215297" denoted GCN 21529 with note 7. Once recovered, all 20 referenced circulars existed and matched their questions. Table 5 reports the results for 8B at 1,024 dimensions.

For 17 of the 18 questions, a correct circular appears in the top five. The single miss (ranked 1,128th) is a labelling error: the labelled circular, GCN 31221, reports an afterglow and a host redshift and makes no statement about absent supernova features, which is what the question asks. On every valid question, therefore, the answer appears in the top five. Two limitations caution against over-interpreting this result. Gemini wrote the questions from circular titles, so they share the circulars' exact wording ("10.4m GTC Ca II doublet") in a way real users will not; and each question lists one or two correct answers where several circulars are often relevant.

Taken together with the earlier evaluation, the results suggest that vectors which rarely retrieve a circular's own event siblings nonetheless retrieve well for questions. What remains untested is questions written by people without the circulars in front of them. We plan 30 to 50 such questions with judged results, drawn from logged searches once search is live and scored by precision at 10; the same evaluation will determine whether hybrid ranking outperforms vectors alone.

Storage and latency

We had estimated 0.4 to 0.6 GB at 1,024 dimensions for the full corpus of approximately 45,000 circulars, assuming a vector index two to three times the size of the raw vectors. With all 13,247 circulars loaded, the table occupies 58 MB and the index 14 MB, 72 MB in total, which projected to approximately 0.25 GB for the full corpus. The measurement after the archive load confirmed it: 234 MB for 44,643 vectors (see Loading the archive). A search consumes approximately 35 request units and 20 milliseconds in the database. Because the index is approximate, we compared it with an exact search over the same vectors: across seven queries, the top result was identical in every case, and 66 of the 70 top-ten results agreed.

Query latency is dominated by the hosted model. Each search first embeds its query with 8B, which takes 0.7 to 0.9 seconds, so caching query embeddings is worthwhile. Because an 8B model cannot practically be self-hosted on our infrastructure, semantic search depends on the availability of a hosted provider. During an outage, semantic search is unavailable while keyword and exact search continue to function. This dependency is the cost of the 12% improvement.

A provider-side fault: invalid and unrelated vectors

During search tuning, identical queries returned different nearest circulars on different calls. Embedding a single query ten times revealed the cause: most calls agreed, but some returned a vector unrelated to the query (cosine similarity of 0.14 against the others), and a few returned NaN. In all cases the API reported success.

• Initial rate. 8 of 400 fresh document embeddings were unrelated to the earlier vector for the same text (cosine similarity of approximately 0.2), as were 2 of 30 query embeddings.
• Faulty vectors resemble one another. Faulty vectors for two different queries had a cosine similarity of 0.85 to each other, suggesting that a faulty backend returns approximately the same output regardless of input. This makes many of them detectable by comparison with a known faulty pattern.
• The 0.6B model is unaffected. It returned no NaN and identical vectors in every test.

Every faulty result in the first measurements occurred while another bulk run was using the same account, so our initial hypothesis was contention between our own requests. A retest with no other traffic from our side refuted it in part. Unrelated vectors fell to zero and valid calls agreed exactly, but the NaN rate varied with time: 30 to 70% of single calls failed in one ten-minute window and 0 of 96 in the next, for queries and circulars alike. We therefore attribute the fault to the provider, most likely to load on its shared GPUs, and conclude that it cannot be prevented from the client side.

Some faulty vectors did reach storage. Scanning the 31,368 archive vectors against the faulty pattern flagged two (GCN 27291 and 25442); re-embedding confirmed both, as their stored vectors were unrelated to two fresh vectors that agreed with each other. Both were replaced. A pattern scan cannot be relied on to catch every case, however, so the only dependable check for a stored vector is to embed the circular twice and keep the result only when both calls agree.

Mitigation: agreement between two calls

Because the API reports success in either case, there is no error to catch, and the value itself must be validated. Every circular is now embedded twice, and its vector is retained only when the two calls agree at a cosine similarity of at least 0.95; healthy repeats agree at 0.996 or above, while faulty ones fall near 0.2. A disagreement triggers a third call, and the agreeing pair is kept; if no pair agrees, nothing is stored and the next run retries. The check applies to both the bulk runs and the live sweep. It doubles the number of embedding calls, a negligible cost at $0.01 per million tokens.

A limitation of agreement

Agreement alone proved insufficient. During one period of provider faults, 6 of 12 calls for a single query returned the identical faulty vector: because the faulty backend produces nearly the same output for any input, two faulty calls agree perfectly and pass the check. Valid vectors for short queries, conversely, differed from one another more than expected (cosine similarity of 0.74 to 0.87), so two valid calls frequently failed the 0.95 threshold, and approximately one fifth of live queries fell back to a degraded search.

Faulty vectors nonetheless form a recognisable family. Faulty vectors from different inputs have a cosine similarity of 0.5 to 0.86 with one another, whereas real vectors reach at most 0.36 against them (the 99.9th percentile over 44,582 circulars). A vector within 0.45 of a stored faulty example is therefore now rejected outright. With faulty output excluded in this way, two query calls need only agree at 0.6, and three are sent in parallel from the start. Measured during the same period of faults, all 10 of 10 queries obtained a usable vector, at an average of 3.6 calls and 1.4 seconds, compared with approximately one pair in four before the change. The same comparison found 9 further faulty vectors in the cache, including GCN 8839 and 41339, which had appeared as unrelated neighbours of an earlier faulty query: faulty queries had been matching faulty circulars. All nine were re-embedded and confirmed.

Verification of stored vectors

Rather than re-check all 44,615 stored vectors, we re-embedded a random sample of 15,000 and compared each with its stored vector (Table 6). One stored vector (GCN 15087) was faulty and was replaced; two circulars remained unresolved because no two fresh calls agreed, and two fresh calls were themselves faulty.

The rate of faulty stored vectors is therefore approximately 0.007%, which implies two or three among the 29,600 unchecked and does not justify re-embedding the remainder; those were instead scored against the faulty fingerprints. The fault is real but rare in what was stored; the greater risk lies in fresh calls that are not checked at all, which is where the safeguards above apply.

Maintaining the index

New circulars are embedded by a periodic sweep rather than by a hook triggered when a circular is saved. Every 15 minutes the sweep queries the database for circulars from the preceding two days that lack a vector, embeds them, and stores the results. We chose this design because a fire-and-forget hook previously caused another component of the system to lose work silently for several weeks: when a call failed, nothing recorded that it needed to be retried. A sweep cannot lose a circular, since any circular missed by one pass is found by the next.

Embedding the same text twice does not always yield an identical vector. A circular embedded by both the live path and the bulk run produced vectors with a cosine similarity of 0.997 rather than 1.0. Hosted models batch requests together on the GPU, and the floating-point arithmetic depends slightly on batch composition. The difference is far too small to alter a search result, but it means that two runs cannot be compared by hashing their vectors, and it is why the agreement check uses a similarity threshold rather than exact equality.

Loading the archive

The archive is the principal motivation for this work: approximately 31,600 GCN Circulars from 1997 to 2024 that reside in our email store but had never been extracted. We embedded them before extraction, so that the complete record became searchable first and acquires its structured data later.

Keys. Each vector is stored under the key that extraction will later assign to its circular (GCN-<number>), so that the two meet without a migration. We derived the keys with extraction's own parser rather than from email identifiers, and this exposed a defect: GCN contains two circulars numbered 18448.5 and 18453.5 (both concerning the gravitational-wave candidate G194575, October 2015), which the parser truncated to 18448 and 18453, the keys of two different circulars. The parser now retains the fraction. Since 239 archive circulars had already been extracted and embedded, 31,368 remained.

Embedding and verification. The run embedded 31,356 distinct texts (12 were exact duplicates), 13.8 million tokens at a cost of approximately $0.14, in just over two hours at the reduced request rate and with no contention backoffs. Cleaned text averaged 2.79 characters per token. Before loading, the stored vectors were checked against the provider fault, first through the random sample described above and then by scoring every vector against the faulty fingerprints. Together these found 11 faulty vectors among 44,615, all of which were re-embedded and confirmed.

The load. A 100-row pilot measured the cost at 2,087 request units, approximately 21 per row, compared with 31 in the first load. The full load followed in 313 statements of 100 rows (Table 7).

Storage. After the load, the table and its vector index share a single storage range of approximately 234 MB for 44,643 vectors, or roughly 5.2 KB per vector. At 13,247 circulars the two occupied separate ranges in a ratio of approximately 80:20, which suggests a split near 187 MB and 47 MB, although only the total is measured. This matches the projection of approximately 0.25 GB made at 13,247 circulars and is well below the original estimate of 0.4 to 0.6 GB: every circular since 1997 occupies a few percent of the 10 GiB free tier.

A correction that silently did nothing. Four of the faulty vectors were already in the table, so a small correction file was written to replace them. It reported INSERT 0 0. The load's upsert rewrites a row only when the circular's text, model or stored size has changed, which keeps repeated runs inexpensive; a corrected vector for unchanged text alters none of these, and the row was skipped without error. The four rows were updated directly, and the export script gained an option to force vector-only corrections.

Acceptance on archive-only questions. We tested two questions that only the archive can answer. For "Was S190822c a real gravitational-wave event?", the top result was the retraction, GCN 25442, which reports that after human inspection the trigger "is no longer considered to be a candidate of interest due to a data quality issue in the LIGO Livingston detector". For "What redshift did X-shooter measure for GRB 221009A?", retrieval returned the X-shooter measurement of z = 0.151 (GCN 32648) together with the later, consistent GTC spectrum (z = 0.1505, GCN 32686). Archive circulars are not linked to events until extraction runs, so both results came from the vectors alone.

Conclusions

• Clean the text before embedding it. Tables and author lists dominate long circulars. Removing them deterministically had a larger effect than any choice of model, and a report verifying that no prose was lost made the step safe to apply.
• Evaluate on the target corpus. The published benchmark scores of the two models are close; on our circulars, 8B was 12% better and corrected precisely the queries of greatest interest.
• Treat model and stored size as separate decisions. With a truncatable model, 1,024 dimensions retained 99.7% of the full-size score at a quarter of the storage, and 99.4% on the full archive of 44,615 circulars.
• Vectors group by report type, not by event. Between circulars they rarely retrieve their own event's siblings, yet for questions they perform well: the first benchmark placed a correct circular in the top five for every valid question. Exact name matching remains for event identity and as a fallback.
• Validate every value, not only every response. A hosted model can return NaN or unrelated vectors while reporting success. Requiring two calls to agree is an inexpensive safeguard but not a sufficient one, since faulty calls can agree with each other; rejecting vectors that resemble known faulty output closes that gap. Inputs require the same scrutiny as outputs.
• Ground truth requires validation as well. The generated benchmark had footnote numbers fused into its circular identifiers, and one label pointed to a circular that does not address its question.
• Confirm that a write took effect. An idempotent upsert keyed on the input skipped a corrected vector for unchanged text without reporting an error.

Future work

With all 44,642 vectors loaded, the immediate priorities are to extract the archive, so that its circulars are linked to their events and gain structured data; to monitor the embedding sweep, since a stopped sweep silently drops new circulars from search after two days; and to construct a benchmark of questions written by people, drawn from the search log, together with a cache of query vectors.