Sharing the Technical Details on Genomic Large-language

Sharing the Technical Details on Genomic Large-language "Toy Model"

This is a follow-up of yesterday’s post titled “High-schooler Builds Test Program to Understand Genomic AI Papers”. I like to share information on how to train and run this program, and some of the findings.

Why is it Called S4D?

This is a direct implementation of the math from the Stanford/Arc Institute group building evo/evo2. S4 here stands for “Structured State Space sequence model” and D refers to “diagonalization”.

The dominant architecture for large-language models in based on transformers, and that is what you see implemented in popular programs like chatGPT, Claude, Google’s BERT/Gemini or even Alphafold. The Stanford group working on evo/evo2 came up with a different mathematical formulation to overcome some limitations of the transformer architecture. I discussed the improvements in this 2025 blog post.

How can you run this program?

Guhan wrote the instructions on how to train and test this model in the README file.

The default code looks for training data here “data/plastid/refseq_full/refseq_plastids_all_clean_no_n.fna.gz”, but you can use “–fasta-file” to pick a different FASTA file.

To train, you run -

python run_plastid.py \
  --preset full-recovery \
  --prediction-unit triplet \
  --output-dir outputs/my-triplet-run

To prompt and generate new sequences, you run something like -

python generate_dna.py \
  --checkpoint outputs/my-triplet-run/best_loss.pt \
  --prompt ACGTTGCAACGTTGCA \
  --max-new-bases 120 \
  --decoding-mode sampled \
  --temperature 0.8 \
  --seed 13

Key Findings

We are still playing with this code and learn as much as possible. Guhan listed the following findings -

  • Increasing S4D size improved validation loss, but the gains showed diminishing returns. The improvement was much larger from 4.13M to 16.60M than from 16.60M to 26.98M.
  • The matched transformer performed worse than the respective S4D model. S4D exceeded by 3.08% in recursive accuracy.
  • Accuracy was higher in inverted repeats than in LSC or SSC regions.
  • S4D and Transformer differed in repeated-base behavior.
  • The model’s accuracy declined as generation continued.
  • Teacher-forced accuracy is much higher than recursive accuracy.
  • Increasing context length improved performance most from 128 to 256 bases while it plateaued by 512 bases.
  • Performance was consistent across random seeds; recursive accuracy varied by less than 0.24%.

Written by M. //