n8500x/scala2py repository overview

⁠scala2py — static Scala → Python converter, in one image

n8500x/scala2py turns Scala 2 source trees into runnable Python and proves the result: verify compiles and runs the original with the real Scala compiler, runs the converted Python, and diffs the two outputs. Nothing is downloaded at run time, so the image works air-gapped.

docker run --rm -v "$PWD:/work" n8500x/scala2py convert src/main/scala -o out
docker run --rm -v "$PWD:/work" n8500x/scala2py run     out/com/acme/Main.py arg1 arg2
docker run --rm -v "$PWD:/work" n8500x/scala2py verify  src/main/scala --args arg1 arg2
docker run --rm                 n8500x/scala2py selftest

⁠Why not regexes / an LLM prompt

Scala's grammar is too large to pattern-match. scala2py is a small compiler pipeline:

Scala source ─► scalameta parser (JVM, bundled)  ─► JSON AST
             ─► desugar + Python `ast` emitter   ─► valid Python, always
             ─► ruff format
             ─► scala_compat.py runtime shim      (imported as `S` by the generated code)

• A real parser. scalameta 4.17 parses every Scala 2.12/2.13/3 construct; the front end dumps its tree generically, so a scalameta upgrade needs no code changes.
• ast nodes, never string concatenation. Generated files always parse.
• A runtime shim instead of rewriting semantics. S.List, S.Map, S.Option, S.Try, S.PartialFunction, S.Stream, S.Float32, Java toString/% formatting, JVM integer / and % … so the translation stays close to 1:1 and the output prints exactly what Scala printed (List(1, 2), Some(3), 1.0E7, true).
• Library knowledge lives in tables (converter/scala2py/mappings.py, extensible via --mappings your.yaml), not in the emitter.
• Nothing is dropped silently. A construct the emitter cannot translate becomes an S.unsupported("<construct>", "<original source>") call that raises at run time, and is listed in scala2py-report.md with file, line and source — that report is the work list.

⁠Commands

scala2py --help / scala2py <cmd> --help list everything.

Files that are not compilation units — statements at top level, REPL-style snippets — are detected automatically (script mode): they become module-level Python with args bound, run executes them directly, and verify runs the original through the Scala 2.13 script runner.

⁠Examples: database, joins, files

Three realistic programs are baked into the image, verified the same way as everything else (the Scala original runs against the bundled SQLite JDBC driver, the Python against sqlite3, outputs byte-identical):

docker run --rm n8500x/scala2py examples                      # list them
docker run --rm -v "$PWD:/work" n8500x/scala2py examples jdbc-report --to ex   # copy, convert, run

⁠JDBC — Scala

def revenueByCustomer(conn: Connection): Unit = {
  val sql =
    """SELECT c.name, c.city, COUNT(o.id) AS orders, COALESCE(SUM(o.qty * o.unit_price), 0) AS revenue
      |FROM customers c LEFT JOIN orders o ON o.customer_id = c.id
      |GROUP BY c.id ORDER BY revenue DESC, c.name""".stripMargin
  val rs: ResultSet = conn.createStatement().executeQuery(sql)
  println("\ncustomer   city       orders    revenue")
  while (rs.next()) {
    println(f"${rs.getString("name")}%-10s ${rs.getString("city")}%-10s ${rs.getInt("orders")}%6d ${rs.getDouble("revenue")}%10.2f")
  }
  rs.close()
}

def loadOrders(conn: Connection, minQty: Int): List[Order] = {
  val ps = conn.prepareStatement("SELECT id, customer_id, product, qty, unit_price FROM orders WHERE qty >= ? ORDER BY id")
  ps.setInt(1, minQty)
  val rs = ps.executeQuery()
  val out = scala.collection.mutable.ListBuffer[Order]()
  while (rs.next()) out += Order(rs.getInt(1), rs.getInt(2), rs.getString(3), rs.getInt(4), rs.getDouble(5))
  out.toList
}

// in main: join the loaded tables in Scala
val byCustomer = orders.groupBy(_.customerId)
val perCity = for {
  c <- customers
  o <- byCustomer.getOrElse(c.id, Nil)
} yield (c.city, o.amount)
perCity.groupBy(_._1).toList.sortBy(_._1).foreach { case (city, rows) =>
  println(f"  $city%-10s ${rows.size}%2d orders ${rows.map(_._2).sum}%9.2f")
}

⁠JDBC — generated Python (unedited)

@dataclasses.dataclass(repr=False, frozen=True)
class Order(S.CaseClass):
    id: int
    customerId: int
    product: str
    qty: int
    unitPrice: float

    @property
    def amount(self) -> float:
        return self.qty * self.unitPrice


class _OrdersReport(S.ScalaObject):
    def revenueByCustomer(self, conn) -> None:
        sql = S.stripMargin(
            "SELECT c.name, c.city, COUNT(o.id) AS orders, COALESCE(SUM(o.qty * o.unit_price), 0) AS revenue\n        |FROM customers c LEFT JOIN orders o ON o.customer_id = c.id\n        |GROUP BY c.id ORDER BY revenue DESC, c.name"
        )
        rs = conn.createStatement().executeQuery(sql)
        S.println("\ncustomer   city       orders    revenue")
        while rs.next():
            S.println(
                S.format_("%-10s %-10s %6d %10.2f", rs.getString("name"), rs.getString("city"),
                          rs.getInt("orders"), rs.getDouble("revenue")))
        rs.close()

    def loadOrders(self, conn, minQty: int) -> list:
        ps = conn.prepareStatement(
            "SELECT id, customer_id, product, qty, unit_price FROM orders WHERE qty >= ? ORDER BY id")
        ps.setInt(1, minQty)
        rs = ps.executeQuery()
        out = S.ListBuffer()
        while rs.next():
            out += Order(rs.getInt(1), rs.getInt(2), rs.getString(3), rs.getInt(4), rs.getDouble(5))
        return S.toList(out)

    def main(self, args: list) -> None:
        S.Class_.forName("org.sqlite.JDBC")
        conn = S.DriverManager.getConnection("jdbc:sqlite::memory:")
        try:
            ...
            byCustomer = S.seq(orders).groupBy(lambda _1: _1.customerId)
            perCity = S.seq(customers).flatMap(
                lambda c: S.seq(byCustomer.getOrElse(c.id, S.lazy(lambda: S.List()))).map(
                    lambda o: (c.city, o.amount)))

            def _pf1(*_a):
                city, rows = _a[0] if len(_a) == 1 else _a
                return S.println(S.format_("  %-10s %2d orders %9.2f", city, S.size(rows),
                                           S.seq(S.seq(rows).map(lambda _1: _1[1])).sum))

            S.seq(S.seq(S.toList(S.seq(perCity).groupBy(lambda _1: _1[0]))).sortBy(lambda _1: _1[0])).foreach(_pf1)
        finally:
            conn.close()

Both sides print exactly this:

loaded 4 customers and 6 orders

customer   city       orders    revenue
Initech    Dubai           2     161.50
Globex     Abu Dhabi       2     129.98
Acme       Dubai           2      84.97
Umbrella   Sharjah         0       0.00

per city (orders with qty >= 2, joined in Scala):
  Abu Dhabi   2 orders    129.98
  Dubai       3 orders     97.47

top customer: Globex (129.98)
customers without qualifying orders: Umbrella

S.DriverManager speaks jdbc:sqlite: out of the box; any other database is one line — S.DriverManager.register("jdbc:postgresql:", lambda url, user, pw: psycopg2.connect(...)) — and the ? placeholders are rewritten for %s-style drivers.

⁠In-memory join — Scala → Python

val lines = for {
  s <- stock
  p <- productBySku.get(s.sku)          // inner join (Option)
  w <- warehouseByCode.get(s.warehouse)
  sold = salesByKey.getOrElse((s.sku, s.warehouse), Nil)   // left join
} yield Line(p, w, s.onHand, sold.map(_.qty).sum, sold.map(x => x.qty * x.price).sum)

def _f1(s):
    def _f2(p):
        def _f3(w):
            sold = salesByKey.getOrElse((s.sku, s.warehouse), S.lazy(lambda: S.List()))
            return Line(p, w, s.onHand, S.seq(S.seq(sold).map(lambda _1: _1.qty)).sum,
                        S.seq(S.seq(sold).map(lambda x: x.qty * x.price)).sum)
        return S.seq(warehouseByCode.get(s.warehouse)).map(_f3)
    return S.seq(productBySku.get(s.sku)).flatMap(_f2)

lines = S.seq(self.stock).flatMap(_f1)

⁠CSV in / out — Scala → Python

def writeCsv(file: File, header: String, rows: Seq[String]): Unit = {
  val out = new PrintWriter(file)
  try { out.println(header); rows.foreach(out.println) } finally out.close()
}
def readCsv(file: File): List[Array[String]] = {
  val src = Source.fromFile(file)
  try src.getLines().drop(1).map(_.split(",")).toList finally src.close()
}

def writeCsv(self, file, header: str, rows: list) -> None:
    out = S.PrintWriter(file)
    try:
        out.println(header)
        S.seq(rows).foreach(out.println)
    finally:
        out.close()

def readCsv(self, file) -> list:
    src = S.Source.fromFile(file)
    try:
        return S.toList(S.seq(S.drop(src.getLines(), 1)).map(lambda _1: S.split(_1, ",")))
    finally:
        src.close()

The output is deliberately a faithful, runnable translation rather than idiomatic Python: S.seq(...), S.format_, S.lazy(...) carry Scala's semantics so the program behaves identically on day one; the differential harness then lets you simplify safely.

⁠What the test-suite proves

selftest runs 42 tests. The heart of it is differential: every program is compiled + run by scalac and converted + run by Python, and stdout must be byte-identical (exit codes too).

• tests/programs/ — 9 synthetic programs, ~750 lines, written to cover the syntax that breaks naive converters: curried/default/by-name/vararg params, @tailrec → loops, JVM / %, f"..."/s"..." interpolators, while/do/for-comprehensions with guards and value bindings, lazy vals, sealed hierarchies + nested extractor patterns, guards, alternatives, ::/+:/List(a, _*) patterns, type patterns on Any, case lambdas, partial functions, traits with super, abstract classes, companions with apply, operator overloading (+, unary_-), Ordered, generics, anonymous classes, custom exceptions, try/catch/finally as expressions, Try/Either/Option combinators, sys.exit, StringBuilder, stripMargin, raw strings, mutable collections…
• tests/realworld/ — two GitHub projects copied verbatim (licences included):
  • stettix/scalisp⁠ — a Lisp interpreter / REPL (public domain). Fed program.lisp on stdin: 54 lines of output, identical. This one drove auxiliary constructors, arity overloads, extends HashMap, this += …, +: patterns, (z /: xs)(op), PartialFunction.lift/orElse, Float arithmetic and #:: streams.
  • TheAlgorithms/Scala⁠ — 22 files (MIT) + a test driver that exercises every routine with deterministic inputs. 100 % coverage, identical output.

Both convert at 100 % with no unsupported construct — see tests/realworld/*/SOURCE.md.

• tests/scripts/ — a top-level-statements script, diffed against the Scala script runner.
• examples/ — the three bundled examples above, including the JDBC one against SQLite.
• The closures and orderings programs are regression fixtures for the breadth run below.

⁠Breadth: 1,287 Rosetta Code programs

To answer "is it comprehensive?" honestly, dev/breadth.py runs the converter over every Scala solution in the Rosetta Code data mirror — 1,287 files, 955 tasks, 36,830 lines, many authors, Scala 2.8 → 3 — and compiles/runs a random sample both ways. Five fix rounds were driven purely by that report (full table and the remaining limits in tests/realworld/ROSETTA.md⁠):

Two-thirds of what is still reported as unsupported are imports with no Python mapping (jline, Swing, java.time, …), i.e. library shims, not syntax.

⁠Reading the report

scala2py 0.1.0: converted 23 file(s) -> out
  100.0%  Sort/HeapSort.scala
   89.2%  net/janvsmachine/ScaLisp.scala  (7 unsupported)
mean coverage 99.5%  — details: out/scala2py-report.md
  main: driver.AlgoDriver  ->  scala2py run out/driver/AlgoDriver.py

Coverage = share of AST nodes that were translated. scala2py-report.md ranks the unsupported constructs by frequency and lists every occurrence with line and source. Work the list top-down: 30-odd constructs cover ~90 % of ordinary Scala.

⁠Extending it (the loop)

1. Put the failing Scala in a fixture under tests/programs/<name>/ (or point at your repo) and run the dev loop — it mounts the working tree over the image, so there is no rebuild per iteration:

   dev/verify.sh patterns                       # a bundled program
   dev/verify.sh /src/tests/realworld/scalisp --stdin /src/tests/realworld/scalisp/program.lisp
   

   Copy
2. Fix it in the right layer:
   • a library call → a table entry in mappings.py (NAMES, PATHS, DISPATCH, SYMBOLIC_METHODS, EXCEPTIONS, LIBRARY_BASES, …) or a method on the runtime classes in converter/scala2py/runtime/scala_compat.py;
   • a syntax form → an e_<NodeType> method / pattern case in emitter.py (scala2py ast file.scala shows the node shape).
3. Keep selftest green; rebuild with build.ps1 (runs the suite before tagging).

Project-specific library mappings go in a YAML file passed as --mappings:

names:   { Logger: "logging.getLogger" }            # bare identifiers
paths:   { "Instant.now": "datetime.datetime.now" } # dotted paths
imports: { "com.acme.util.Json": "from acme_json import Json" }  # Scala import → Python import line

Code that is expensive to automate is better flagged for a human: heavy implicits, type-level programming (shapeless, cats/ZIO effect stacks), macros.

⁠Known limits

• Syntax only, no type checker. Types are inferred heuristically (declared types, literals, a project-wide member/parameter type table). Where the operand types of + are unknown the code calls S.plus, which decides string-vs-number at run time; Int/Double are distinguished when declared.
• Implicits are not resolved; implicit parameter lists become keyword-only params defaulting to None (Ordering[T] params default to the natural ordering). Implicit conversions are not applied. The one slice that is handled: an explicit Ordering argument (sortBy(f)(Ordering.Int.reverse), sorted(Ordering.by(...))) and an implicit val … : Ordering[T] in scope when PriorityQueue[T]()/TreeSet[T]() is created with that type argument.
• Reference equality (eq/ne → is) on strings and numbers follows Python interning, not the JVM; XML literals and parser-combinator extractors are unsupported; >>> guesses the integer width from the value.
• Overloads / auxiliary constructors are supported when they differ in arity.
• Spark/PySpark mapping exists in the tables but is untested in this version.
• Scala 3 parses (--dialect scala3); enum and given/using are only partially translated.
• Generated code needs Python ≥ 3.10 (match statement) and scala_compat.py, which convert copies next to the output. It is correct-first, not idiomatic-first: collections are S.List(...) rather than list comprehensions. It is a starting point for a human/LLM clean-up pass, with the differential harness as the safety net.

⁠Building the image

.\build.ps1          # build, run selftest, tag YYYYMMDD-HHMMSS + latest
.\build.ps1 -Push    # also push (only after the image has been verified)

Stages: Maven shades scalameta + the Scala 2.13 compiler into scala2py-frontend.jar; the runtime image is python:3.13-slim with a copied Temurin 21 JRE, the converter installed as the scala2py entry point, and the test-suite under /opt/scala2py/tests. If your host intercepts TLS, the interceptor's CA can be passed as the BuildKit secret extra_ca (build.ps1 does this automatically); it is used only while Maven/pip download and is never written to a layer.

⁠Layout

frontend/           Maven project: scala2py.Dump (scalameta → JSON Lines) + shaded scalac + sqlite-jdbc
converter/scala2py/ cli.py, frontend.py (JVM driver), emitter.py, mappings.py,
                    runtime/scala_compat.py
tests/              test_programs.py (differential), test_runtime.py, test_examples.py,
                    programs/, scripts/, realworld/ (SOURCE.md per project, ROSETTA.md)
examples/           jdbc-report, collections-join, csv-pipeline
dev/                verify.sh (mounted-tree dev loop), breadth.py (corpus measurement)
Dockerfile, build.ps1, hub-overview.ps1

Inside the image the same tree lives under /opt/scala2py (scala2py docs --to DIR copies it out), so the published image is self-contained: converter, runtime, tests, examples, docs.