Contributing to Swift: From Proposal to Shipped

Even mere mortals like us can make contributions to the Swift open source project. You don’t have to be a compiler expert or even know C++. There is a lot of Swift code in the Standard Library, Package Manager, Foundation, and Dispatch. In this talk, Russ Bishop covers the basics of cloning the repos, making builds, and getting a custom Xcode toolchain working. Then he walks through how he implemented and shipped three Swift Evolution proposals in Swift 3. Once you see how easy it is, you’ll want to do it too!

---

Introduction (00:00)

My name is Russ Bishop and today I am going to talk about contributing to Swift.

This is the story of how I decided I wanted to get involved in contributing to the Swift project. But I had never done anything like that before; never contributed to a compiler or standard library or development tools. I ended up shipping which is in the Swift 3 or the Xcode 8β these four evolution proposals. None of them are huge or life changing, but it was fun to do.

My theory is if I can take some of the burden off of the Swift core team by employing some of these smaller proposals then maybe they can focus on much more important things that I am not qualified to do, supporting opening existentials or redesigning clutch protocols or whatever the case may be. I want to encourage that you can do it too: it is not magic or genius, it is a bit fiddly and takes a bit of work but you can definitely do it.

Swift.org (02:10)

I want to encourage everyone here, give bugs.swift.org a quick look, find a starter bug, see if you can work on it. Or maybe some of the evolution proposals that have not been implemented yet. Give it a try.

I am going to go over a few basic things about how to get started and this will be useful even if you want to look at the code or see how the standard library is implemented. Because there is a lot of interesting things in there.

• The master branch does require Xcode 8. If your toolchain is set to Xcode 7.3, you will be sad.
• It asks you to create sort of a parent directory, but then to clone Swift repo into a sub-directory of that parent directory. The script that will do all the auto checkouts see these out here into your Swift source directory and then it checks out LLVM, clang, the Swift core foundation, all the other associated libraries, and the tools that are necessary.
• You can see here there is a – clone SSH I believe. If you are set up to use SSH with GitHub that will work.

This gets you set up with all the stuff checked out. It will take a few minutes, but it is not too bad.

Contributing (03:25)

If you do want to make contributions you go to, for a particular project, you can definitely do, which I made the mistake of doing originally, was I renamed origin to upstream, and then I added my GitHub fork as origin. The problem is there is an update checkout script that updates all these projects and then it gets very confused and things do not get updated and it creates a big mess. I recommend adding your fork on GitHub as a separate repo in the get configuration and then you can manually when you set up your branches set them up to push to your fork. But otherwise that leaves master and the local repo pointing to the actual Swift official sources (Internet, git docs guide, and ask another git expert).

Eventually, when you get ready to contribute, you can open PR against the Apple Swift repo, link the JIRA bug and await review. All the core team is nice; sometimes criticisms can be on point and they have many issues, but they are always nice and thankful for the additional contribution.

Building (05:07)

Step 1: I want to build a Swift compiler. How does that work?

There is a script in the utilities subdirectory, ./utils/build-script. There are couple of flags: -R which gives you release mode (if you do not specify that, you will get debug mode), -c for clean, -x to generate an Xcode project (do not get very excited about that; I will tell you why below).

The best way to learn about the entire build system, there is a build-presets.ini file in the utils directory that specifies all the different high level actions, and what each action is composed of in terms of the build (actions test, validation test, test macOS, test iOS, build iOS, build tvOS, etc). You can create your own presets and your own .ini file.

This is probably the fastest way I have found to do a complete build from scratch, which is release mode for LLVM / Clang but -x debug Swift (you can still debug the Swift compiler). 33 minutes on my 2015 MacBook Pro - it takes time (that is probably the worst part about contributing). Fortunately the build system does do incremental builds. If you make a change (e.g. like I did to add a function), it will rebuild sequence - it does not have to rebuild the compiler and the standard library (that can be helpful).

Why not to get excited about the Xcode project? It can only build for macOS and none of the standard library or any of the Swift source files are available: they are templates that are dynamically generated during the build. When you browse in the Xcode project you will see red files that are missing. When you do a build they will appear but if you modify them your changes will immediately be overwritten by the next build. It can be a very frustrating experience. I found when working on the standard library I used a plain text editor, I do not even use Xcode. This does work great if you want to work on the compiler’s C++ sources. We will see if we can do something about improving that in the future.

Directory Structure (08:07)

Where to find things in the directory structure? There are multiple subdirectories… it took me a while to find my way around!

The /include & /lib have the compiler C++ sources (that is the actual compiler). If you want to mess with /tools/SourceKit, the full build of SourceKit, all of it is there ready to use. You have the full access to the compiler, and everything the compiler knows about the code is available to you. Also, SourceKit builds cross-platform now (it is not only on macOS).

The standard library is in /stdlib, as you might expect. Almost entirely written in Swift; there is a bit of C++ code hiding. The /stdlib/private includes a standard library for the unit tests. The /stdlib/public is almost the entirety of the actual standard library.

A cool directory to check is /sdk, that has all the stuff that says if I have CGContext and now suddenly in Swift there is a bunch of properties and methods functions, that is where all that is implemented. There are a bunch of sub-directories under /sdk for all the different frameworks: Core Audio, UIKit, etc., that have all the Swift niceties in there.

/stdlib/public/runtime (event horizon reference, where we are going we do not need eyes to see): Interesting but dangerous code in that directory. If you are curious about how it works (e.g. how Swift and Objective‑C interoperate, their objects, how can you inherit from an Objective‑C object in Swift), there is code there that deals with that. There is also in libobjc, the actual Objective‑C runtime is open source (Apple posted it on opensource.developer.apple.com), it runs on Linux, Windows, and various platforms. There is a bunch of Swift-specific stuff in there as well. You can find out how weak references are implemented and all that stuff.

The /apinotes, that is where all the weird type renaming, mapping selectors or marking things not available - all that stuff that happens when Objective‑C objects are imported into Swift, that is all described in the API notes. In theory you could have API notes for your own frameworks.

/docs: Again I will say do not get excited about what is in docs. There is one concurrency document that describes awesome stuff about a potential concurrency model where Swift could, in the type system, enforce that types can only be handed across thread boundaries if they implement a copying protocol, and how you can build on top of these primitives to get message passing and async/await, etc. That was the core team’s fever dreams… and is not on the roadmap anytime soon. Everything that is happening now is all going through Swift evolution. Also, some of it is out of date (e.g., the generics document). I like it because it gives you a historical view of what the team went through as they were developing this… but be careful with docs.

/unittests: These are not the actual unit tests. There are a couple of the C++ compiler tests around the parser; otherwise, there is not much in that directory. The actual unit tests are in the test directory and they are almost entirely in Swift (even tests for core compiler features).

There is a /test/l_stdlib directory: helpfully named so that standard library test will run first. ¯_(ツ)_/¯

/utils: Build scripts, .gyb, etc (we will talk about in a minute).

/validation-test is the massive test suite of regression test cases, validating shapes of types, if I have a custom SequenceType that properly inherits the default implementations it should and things of that nature. If you are going to contribute to the standard library, particularly this one /test/l_stdlib and /validation-test is where your tests will go. I mostly say this because my very first PR I opened I put everything in test and then got a nice correction from Demetry: a bunch of this belongs in validation tests. And I had to update my PR.

Generate Your Boiler Plate (13:55)

The thing that frustrates the abilities Xcode: a bunch of the tests have GYB. Swift does not have a macro system currently; there are various language limits in terms of generics that make some things more boiler-platey than we would like them to be.

The compiler’s answer to that is: generate your boiler plate. It is a Python script that processes templates (php, asp, Handlebars, any number of templating engines you might have used in the past), some Python code and anything that is not the Python code is echoed into the file. It has a few basic things: substitutions with the $ sign; the Python code blocks are delimited by %{ … }%; single-line Python is a single %.

This is the reason that the Xcode project does not work because much is generated dynamically by those templates. You can invoke it manually from the command line, pass it a GYB file and it will spit out what the output is: gyb CollectionAlgorithms.swift.gyb > CollectionAlgorithms.swift

This is real code:

% for mutable in (True, False):
%   Self = 'UnsafeMutablePointer' if mutable else 'UnsafePointer'

public struct ${Self}<Pointee>
  : Strideable, Hashable, _Pointer {

%   if mutable:
  public init(allocatingCapacity count: Int) {
    //… }
%   end

}
% end

This is how UnsafePointer is implemented in the standard library; because there is a mutable and an immutable variant, they did not want to have to duplicate all that code generated in a GYB file. For each case (mutable and immutable), it creates this Self variable and you can see how with the ${Self} we are substituting the correct name. Then there will be common implementation (I have left out), if this is the mutable variant in the mutable initializer.

If you want to learn more, there is hundreds of GYB files (not my idea of a fun time!).

Tests (16:00)

There is a build sub-directory when you do a build under that top-level Swift source directory. For every single configuration you have, it generates another sub-directory and puts all the artifacts in there as you are doing builds. If you switch between debug and release, or you build a toolchain and then you change those flags, you end up with of artifacts under all of these sub-directories (I think I consumed 150 gigabytes of data before I realized what was going on).

If we are using Xcode, we have Xcode, the XCTest will launch, run the unit test. What is the compiler using? It is using LLVM’s lit, a very simple utility again written in Python. Read some config files and then it scans the directory looking for tests and then compiles and executes them. Running unit tests alone without the validation test takes about 13 minutes. Be prepared for that. I have created a GitHub gist file that has some scripts that can run individual unit tests that will make things way faster.

Validation tests take over an hour. I do not even think it is a requirement to run them before you open a PR because they let the build machines handle that.

To run a test you can use t (lowercase t is the unit test, uppercase T is the full validation test suite). Make sure that these build settings match whatever you did when you ran your last build otherwise it will go this is a different configuration and then I am going to do a build from scratch and you will add a half our to your unit test run.

LLVM_DIR="/<path>/swift-src/llvm" SWIFT_BUILD_DIR="/<path>/swift-src/build/<build-config>/swift-macOSx-x86_64" run-validation-test() {
    $LLVM_DIR/utils/lit/lit.py -a --param swift_site_config=$SWIFT_BUILD_DIR/validation-test-macOSx-x86_64/lit.site.cfg $1
}
run-test() {
    $LLVM_DIR/utils/lit/lit.py -a --param swift_site_config=$SWIFT_BUILD_DIR/test-macOSx-x86_64/lit.site.cfg $1
}

If a test fails, this is some bash stuff for the .bashprofile or .bashrc. You can set up these parameters and then create bash functions that you can say something, e.g., run test. You do not even have to give it the full path, you can say one /test/l_stdlib slash and the file and it will go run, compile that file, and run all the tests that are inside of it. This saves a lot of time (13 minute unit test run down to 10 or 15 seconds).

Validation tests are handy: when they fail they give you a nice command.

/some/long/path/a.out
  --stdlib-unittest-in-process
  --stdlib-unittest-filter "test/path"

Custom Toolchains (18:59)

The toolchain is that complete package of tools (Clang, LLVM, Swift compiler, linker, codesign utilities, etc.) that allow you to do builds and interact with Xcode. It includes SourceKit. Xcode understands how to use different toolchains, and you can build your own toolchain. There are scripts in the Swift repo that will build a custom toolchain. You can even build packages that you can redistribute to other people and they work as the ones you can download from Swift.org. Somebody else can double-click it, install, and then boom 💥, they have your custom toolchain.

The default Xcode toolchain is inside the Xcode app bundle, Developer/Toolchains/XcodeDefault.toolchain. If you download ones or do your own custom toolchains, those are installed /Library/Developer/Toolchains/*. In the Xcode menu there is Toolchains, and you can see the list of the toolchains and switch between them.

As of Xcode 8β2, you can now use custom toolchains in playgrounds. You can associate a toolchain with a project. That is how that feature of using a project that is on Swift 2.3 or on Swift 3 and Xcode 8 works in the project file it says use this toolchain and that project files says use that toolchain.

The script that comes with Swift that does a full toolchain build, builds for all platforms and it runs all validation tests for all platforms. It takes a long time. Instead, you can do a custom ini file with your own presets and skip platform, skip tests, etc. And but you can still produce a full installable package for distribution. You can even say only build macOS because I want to test something quick and that toolchain works totally fine as long as you know the playground or the Xcode project is macOS and is not trying to use tvOS or iOS.

I did symlink my toolchain into the system location that I can run my script to generate new toolchain and then Xcode is immediately using that updated toolchain. Do not have Xcode open while you are doing that (Xcode will get very angry). But this updates SourceKit with all the quick help completions, auto-completions, and syntax highlighting. It all picks up those changes immediately when you do that because Xcode knows to go find the appropriate version of SourceKit that is in that toolchain directory.

Coordination (22:14)

You may say, this sounds awesome, I want to try my hand at this - where do I go, what do I do?

swift-evolution. If you follow the mailing list, you know it is a very high volume. My advise is avoid the bikeshedding threads. It is nice that everyone can contribute, but I cannot keep up with that: I cannot be mailing completion list on Swift evolution, there is too much traffic. But it is nice if you have some proposals, or you want to read some of the back and forth.

But the day to day development, swift-dev makes total sense. If I ran into a situation implementing one of the evolution proposals where another API that had depended upon had changed in the mean time and it was not possible to implement the evolution proposal the way it was originally written up, some of the core team responded, we can do this.

Then of course bugs.swift.org. If you are going to implement a evolution proposal, or you are going to fix a bug or anything and you are like I do not want to be in a situation where I am implementing something someone else is implementing, this according to the core team, I asked, bugs.swift.org is the place to go. You can search for the evolution proposal number, SE-_whatever_, and if there is not a bug for that evolution proposal create one and assign it to yourself and that is the way you tell the world, “I am working on this particular issue.”

Future Directions (23:58)

I would like to get more involved in the compiler and how to implement some of these in terms of languages (How would I go about editing a parser? How would I add a new keyword?).

I want to recommend the LLVM tutorial: it is implementing a toy language called Kaleidoscope, and it walks you through doing a little parser, creating an abstract syntax tree, and turning it into a REPL and having LLVM emit some code. Then getting some magic for free, saying I want these optimization passes from LLVM where it can do constant folding and loop unrolling etc. A couple of lines of code and LLVM will do all that automatically for you.

In generating a binary that you can execute and adding things, variables and loops and control structures. Jordan Rose has “So you want to be a compiler wizard”. He points out that at least one member of the Swift compiler team has no formal computer science education but is a competent compiler engineer. You do not have to have studied it in school or be super genius: less about genius, more about hard work.

Lastly, I intend to keep working on bugs and implementing Swift evolution proposals. There are new evolution proposals all the time. There is plenty to do, new bugs to fix. Do not be scared if you are not familiar with C++, or have not worked with compilers before: pick something in standard library and work on that. That is all almost entirely Swift code; it is something everybody in this room can accomplish.

Q & A (26:16)

Q: Great presentation. You mention that it takes forever to build if you have a MacBook Pro. I have a MacBook Air.

Russ: My condolences. 😨

Q: You also mention that it is building all the runtimes. But Swift also works on Linux. I am wondering how viable it would be to spin up a big AWS instance and do your compiling there even if you were planning on, even if you were mainly interested in support for the Mac.

Russ: Yes, you can do that and it works fine. The state of Swift on Linux is more mature than I thought last time I checked. The only thing you have to look out for is that the Foundation that is used on Linux is the Swift core libs Foundation, not NSFoundation. There are still some things that are unimplemented but that maybe something that if you are interested in doing, contributing, go in there and you run across one of those things then implement it and submit it as a PR. But yes you can totally do that. There is support for dist cc in terms of building, clang and LLVM. I was not able to fully get it to work, but in theory if you have two or three MacBook Airs you could gang them together and distribute the compilation across multiple nodes, but I was not able to get that to work so I cannot tell you any more about that part.

Q: But the support that exists now, do you think that would be sufficient for using a Linux box to build the Mac toolchain? This may be a stupid question, maybe that is not. Compiling platform. I have not tried that but as far as I know it should work because LLVM has that capability.

Russ: I do not see any reason why you would not be able to but I have not tried it, I do not know. It may be a matter of the SDK not being present on Linux because it does use the system when it builds all of the Objective‑C library stuff and when you are referencing like Apple frameworks, Objective‑C frameworks it is looking at the system framework headers for that information that may be the only thing I can think of.

Q: How much time would you say you spent going through stuff and reading up on things before you started to get comfortable and get a feel for the whole project?

Russ: I attempted to start once and gave up. And then I restarted it if that tells you anything. I think I probably spent over a period of a week I spent you know I do not know how long, hour here and there, a couple of hours one day or another trying to read up, learn more about, and as I ran into problems like the compilation time is taking long, or it takes 13 minutes to rerun one set of tests is there anything I can do and have gathered from tweets and little bits and pieces how to do some of this, which is why I wanted to talk about it because hopefully it can jump start somebody’s efforts. But it was not that long. It did not take very long to jump into it. Especially the standard library, because that is Swift code and writing Swift code everyday it was relatively straightforward.