Building a Real-Time Astronomy Visualization Backend

One day I was looking up at the sky and wanted to know what planet I was seeing. Curious, I tried out some apps that let you point your phone anywhere in the sky and instantly reveal which planets or stars you’re looking at. That blew my mind - how do they do that? I decided to make my own version to understand and build the backend and frontend that powers this kind of astronomy visualization.

The pipeline

I develop on Windows and Linux, and since I was using an iPhone, my only choice was to buy a Macbook to make the app, or just make it a website (I chose the website, obviously). My website sends the observer’s latitude, longitude, and elevation to the backend with each frame. The backend fetches the current sky data from an astronomy API (started with NASA’s JPL Horizons API then switched to the skyfield python library), which returns raw celestial vectors in equatorial coordinates. Those need to be transformed into stable scene-space positions the renderer can place objects at.

The core transform:

v_scene = v̂_api · (d_AU · S)

Where v̂_api is the unit direction vector constructed (on the client) from right ascension/declination, d_AU is the returned distance from the observer to the object in astronomical units (from the API), and S is the global scaling factor you choose for your renderer.

Planet sizing

The backend extracts the raw “angular diameter” of each object in arcseconds from the Horizons API and returns it as-is in the JSON response.

Device orientation

This was one of the most technically challenging parts of the project to figure out.

Modern device orientation APIs on the web are a minefield of compatibility, sensor weirdness, and inconsistent browser support. Most browsers have moved on to WebXR, but for my website, I needed direct, no-drift (or at least minimal drift) orientation control from phone sensors - something “modern” libraries don’t reliably provide (at least from my research, which is weird).

A little history on this is the original DeviceOrientationControls.js (from the Three.js examples) was actually removed from Three.js as of January 2022 because it was unreliable across browsers and has no connection with the evolving WebXR standards. Most devs (and libraries) abandoned it.

I decided to bring back this deprecated library into my implementation. Despite these issues, DeviceOrientationControls was exactly what I needed - a simple way to turn the browser’s alpha, beta, and gamma readings into real, 3D camera control. But I quickly learned the stock library had significant problems like extreme orientation drift, mismatches, odd screen handling, weird offsets, and many other calculations that I can’t give any reasonable explanation for why they were there. So I essentially rewrote the entire thing for my website.

I rebuilt the math for Euler-to-Quaternion conversion, ensuring camera orientation matches the real sky with no gimbal lock or axis confusion. My version detects screen orientation changes, applies offsets for iOS compass quirks, and supports automatic runtime calibration. I added handling for iOS18+ permissions and overlays, so users can reliably enable sensor access. This is a fork/one-off, maintained myself, not reliant on any upstream Three.js updates or browser promises. Reliability for my use case > standards churn.

Source Code

The GitHub repository for this project is coming soon. I want to review the code, clean up anything environment-specific, and make sure it can be run safely and clearly by other people before publishing it.

Conclusion

Building this astronomy visualization website forced me to really have to research and learn everything from coordinate transformations (which was new to me) and planetary ephemerides to the pain of browser sensor APIs and the implementation of rendering 3D scenes in real time. It’s easy to take for granted how seamless these “point-the-phone-at-the-sky” apps seem on the surface - until you attempt to make one yourself and realize how much careful engineering (and, frankly, trial and error) it takes to make something that “just works”.

I still remember all those nights standing on my driveway, laptop on the concrete, phone raised toward the stars, desperately hoping the whole fragile system would work.

Then came the moment, over a month later, the first time I pointed my phone at the sky and the planets actually aligned on screen. I couldn’t help but smile.

By combining a real-time Python backend, coordinate mathematics, and (against all odds) a custom device orientation controls library that actually works, I built something that didn’t just satisfy the fact of creating programs that interact with the real world, but became one of the most rewarding projects I’ve ever done.

Along the way, this project really helped me realize how not everything that’s “modern” or “new” means it’s better, along with how complicated and tricky sensors and spatial computing is. I learned that sometimes the best solutions are the ones that have been around for a while.

Would I do it again? Absolutely. There’s something really special about raising your phone to the sky and seeing the planets and stars light up with labels in real time, especially when you know exactly how every part of the pipeline works under the hood, and that you built it yourself.