Curriculum Progress Intelligence Dashboard: Private API Integration

The portal you click through one student at a time

A franchise education center had about 960 active students spread across its class groups, and the only way to read any of them was the vendor’s own internal portal, one record at a time. Want a student’s current worksheet level, the depth of their grading queue, how many times they studied this month, their recent scores? Open the record, read it, go back, open the next. Nothing exported the full detail an instructor needs, and there was no public API to ask for it.

That is a workable design for checking on one child. It falls apart the moment you try to run a center, where the useful questions are comparative: who has stalled, who is a worksheet away from the next level, whose ungraded pile is growing. All of that data existed. It was just trapped behind hundreds of page loads. My job was to get it into a single view that keeps itself current, and the vendor gave me nothing to build against.

Reading the login scheme out of a minified bundle

Everything started with the login, because if I could not authenticate the way the portal’s own frontend did, nothing else was reachable. The portal runs on OAuth2 password grant, but it never puts the raw password on the wire. The browser hashes it first, and that hashing is where the reverse-engineering earned its keep. I pulled the production JavaScript bundle, worked back through the minified code, and reconstructed the exact routine.

It is two SHA-256 passes with a step that is easy to miss. First it hashes a salt of systemCountryCd + loginId. Then, and this is the part a careless reimplementation gets wrong, it upper-cases the hex of that first digest before folding in the password for the second pass. The result goes out base64-encoded as the grant password. Feed it lower-case hex and it authenticates against nothing.

import { createHash } from 'node:crypto';

// The vendor portal never sends the raw password. Its bundle hashes it in two
// SHA-256 passes, and the easy-to-miss step is upper-casing the first digest's
// hex before the second pass. Lower-case hex 401s on every login.
export function hashLoginPassword(rawPassword: string, systemCountryCd: string, loginId: string) {
const salt = systemCountryCd + loginId;
const firstDigest = createHash('sha256').update(salt, 'utf8').digest();
const upperHex = firstDigest.toString('hex').toUpperCase();
const finalDigest = createHash('sha256').update(upperHex + rawPassword, 'utf8').digest();
return finalDigest.toString('base64'); // sent as the OAuth2 password-grant password
}

Tokens come back with roughly a twenty-minute life. The client refreshes thirty seconds ahead of expiry on a refresh-token grant, and if that refresh is rejected because the session was invalidated, it drops back to a full re-login rather than failing the run. Every API call carries a bearer token and an X-User-ID header, wrapped in a fixed request envelope, and the endpoints themselves are opaque controller codes lifted from the same bundle, things like ATE0010P/GetCenterAllStudentList. Network errors and 5xx responses retry up to three times, and a 401 refreshes the token before replaying the call.

Credentials never sit in plaintext. The vendor password is encrypted at rest with AES-256-GCM, a random IV, the auth tag, and the ciphertext packed into one iv:tag:cipher string, and it is decrypted in memory only at sync time. The dashboard itself sits behind a single-user httpOnly session, so the vendor login and every student record stay server-side.

One roster, a hundred at a time

The sync runs in two tiers. The summary tier is the one that has to be fast, and it pulls the whole roster in around twenty seconds. It logs in, resolves the instructor’s identity and center, loads the master data (the subject list, the class groups, the worksheet catalog), then walks the roster a hundred students per page. Each page’s students are upserted twenty-five at a time so the database round-trips overlap instead of queuing. Active students come first, then inactive, both keyed the same way.

Every write is an upsert on a natural composite key, school plus student, student plus subject, so a run that repeats changes nothing it has already written. That is what makes the whole thing safe to fire again after a partial failure. Alongside the normalized columns, each student row also stores the raw JSON the API returned. If the vendor quietly renames a field, the sync surfaces a mapping mismatch instead of losing data, and the fix is a re-parse of what I already hold rather than a fresh round-trip to an API that might have moved under me. A sync-run table records each run with its status, duration, and any error, so a bad night is visible the next morning.

The deep tier runs per student, on demand. It loops the subjects a student is enrolled in and pulls the parts the summary skips: progress goals, test scores, and the individual study sets down to per-page scores, correction counts, and time against the worksheet’s standard. The full activity history back to 2020 is fetched once, then topped up after that.

A cron that never lines up

Automation runs off a scheduled trigger that wakes the sync endpoint on a fixed hourly cadence. It does not sync everyone every time. It reads which schools are due, syncs those, and reschedules each one to a random point four to six hours out, so the actual sync times drift apart and never settle into a predictable pattern of hits against the vendor. A school with no schedule yet gets a staggered first run somewhere in the next six hours, for the same reason.

The endpoint runs serverless with a hard ceiling on execution time, so it watches a budget. It carries a 250-second guard, and once a run crosses it, it stops and leaves the remaining due schools for the next trigger. Because every sync is idempotent, stopping early costs nothing. When a school’s sync throws, it is not pushed to the back of the four-to-six-hour window. It is rescheduled thirty to sixty minutes out, so a transient failure retries soon instead of waiting for the next full cycle.

6A through O0, flattened to a number

The visible payoff is the progress bar, and it is grounded in the curriculum rather than eyeballed. The vendor’s worksheet catalog lays the curriculum out as an ordered ladder of levels, 6A at the bottom through O0 at the top, and each level carries an absolute-progress offset. At sync time I build an in-memory index over that ladder, one per subject, so any pairing of level code and worksheet number resolves to a single comparable integer: the level’s offset plus how far into it the student has reached.

From there a student’s position is that integer scaled against the subject’s span, clamped to 0 to 100 and rounded to a tenth. The level tag then picks up a color band by stage, cyan for the early levels, blue through the middle, a deeper blue at the top, so a class list reads at a glance. Because the math comes from the vendor’s own ladder, a bar means the same thing for a five-year-old on 6A as it does for a student near the end, which is the entire point of a benchmark.

Outcomes