||5 months ago|
|src||6 months ago|
|.eslintignore||9 months ago|
|.eslintrc.json||7 months ago|
|.gitignore||9 months ago|
|CODE_OF_CONDUCT.md||9 months ago|
|CONTRIBUTING.md||9 months ago|
|LICENSE||9 months ago|
|README.md||7 months ago|
|package.json||8 months ago|
|tsconfig.json||9 months ago|
|yarn.lock||8 months ago|
A quick sample repo of how a rewritten backend in typescript could look using:
In consideration but not shown here:
If you have not already installed typescript as part of working on the hexbear/lemmy frontend or are new to node:
You can follow the instructions on their repo to also get the latest version of node for your OS of choice.
Yarn is the package manager we already use so it's suggested here.
yarn global add typescript
This step is optional for if you want to run migrations/seeders through the knex cli, it's fine to skip it of you dont want to do this step
yarn global add knex
We currently use version 12
|Mac OS X||Enterprise DB|
|Windows x86-64||Enterprise DB|
If you go the docker route, just make sure to update
src/database/knexfile.ts with the correct ip address/port.
(Optional) Install PostgreSQL graphic interface
|pgAdmin||Cross platform GUI tool to manage postgresql databases. This comes with the EnterpriseDB installers as well as the native Linux images. If you're not comfortable with using
|DBeaver||Alternative tool that doesn't come bundled with any of the installers. Highly recommend if you're on Linux.|
Using psql or your GUI tool we want to:
- Create a user
- Create a database
- Make a schema called
publicif it does not exist
- Make a schema called
knex, this wont exist by default. We will store our knex migrations there so they arent affected by operations involving the main schemas.
The above matches what we use in
src/database/knexfile, if you have a preferred local setup then you can adjust the
knexfile. But be warned that the initial migration assumes an empty database with a user
hexbear so adjust accordingly.
To create a migration:
yarn run knex:migration-make migration_name_here
To run all pending migrations:
yarn run knex:migration-run
To create a seeder file:
yarn run knex:seed-make seed_name_here
To run all seeds:
yarn run knex:seed-run
To handle running specific migrations or seeds, the same scripts can be used but you'll need to specify the file name.
knex migrate:(up|down) file_name
knex seed:run --specific=file_name
Starting the App
- Clone this repo and run
yarn run knex:migration-runto build your database
- (Optionally) run
yarn run knex:seed-runto populate your database
- Start the server with
yarn run dev
http://localhost:8080 to play pong or
http://localhost:8080/api/v1/site to see sample data!
VSCode - Has great TS support, autocomplete, is open source (but MS backed), cross platform and one of the more popular editors out there
Postico - OSX postgresql app. Has a slicker UI and is easier to use than pgAdmin but mainly built for querying and lacks management options
DBeaver - Cross platform, good name, probably runs better than pgAdmin
Goals / Why Rewrite / Why Typescript?
The current backend has a lot of Technical-Debt™; we've made several critical changes that have let us scale to 10k+ users and had to put out a lot of fires on the way. These changes have also left us in a state where merging upstream Lemmy is a large job even if we would do it weekly to keep up.
We as a team feel we have hit a dead-end or brick wall with continuing to maintain the current backend in rust as-is while maintaining upstream compatibility. Additionally, there are several architectural changes we would like to make that would both require forking and a significant rewrite of the current codebase.
Since a rewrite is inevitable and rust/choice rust libs have shortcomings or simply aren't ready for maintaining a production load web server, we have decided to use a new language. On the human side, the "core team" feels that the current tech stack is painful or slow to work with, makes onboarding difficult, and has been a turn off from getting new contributors. So-
In the short/mid term we want to:
- Convert majority of client requests to the rest api away from websockets
- Build a v2 api with the goal of splitting the current request/response objects up (ie. make it possible to query smaller data sets so a User component isn't getting irrelevant data, basically make an actual modern/"standard" api)
- Use a mature query builder that allows rewriting the sql queries/joins to be sensible, not reliant on cross joins, can properly filter based on inputs etc
In technical specifics
- Rust compile times are frustrating
- Many Rust libs are not mature enough for a small team web server to be a good idea, nor is the query building/orm side without strict limitations
In human terms
- Some of us are facing severe burnout on Rust or dont find it fun for this project's specifics
- We're finding it difficult to keep contributing fixes because we have to code in a reactionary style and Rust does not lend itself to getting something quick done
- Rust learning curve is steep or the lang itself turns away new contributors
- It offers typing/auto-complete/other goodies to JS
- The ecosystem has many mature libs web servers, middleware, querying, testing, documentation
- Iteration is quick and our BE/FE tech stack would be similar
- We do not face any optimization/performance issues that make compiled langs needed
- Most of us are familiar with TS/JS already
- Learning curve is small and we can maintain a codebase that is friendly to new devs or even someone new to coding
- The TS community is very friendly, welcoming and focused on accessibility. It's extremely fast growing and immensely popular for web projects- As such it offers the best chance to sustain the project via new members
But most importantly, we want this to be fun, even if that means putting in the work to fork and rewrite in a new language. There will be new pain points, new tech debt, and many human hours of work to even bring this to current 1:1 functionality, but that is worth it if the end result is something we own, enjoy and can easily share.
Since this is just a showcase in "how it looks" and "why", we still have to decide on some things:
- What design pattern(s) do we want to use?
- How should we split up work / what areas should we focus on first?
- How do we want the repo structured?
- What other libs do we need?