Few cents about my commits

RoboVM compiler: out of memory -- huge class case

|

RoboVM compiler is quite memory hungry by its nature: creates and manipulates a bunch of strings, have to keep parsed classes structures while compiling etc. Also there is a bug opened #150. Usually its enough to give JVM bigger heap (-Xmx4g) to make everyone happy.
There was a report on gitter channel about OOM that happens on a single file that can’t be fixed by increasing heap size. File just contained about 8K static final string fields.

To reproduce the case dummy file with 8000 strings was generated:

public class DBkeys
{
    static public final String T0001 = "";
    static public final String T0002 = "";
    // lot of more these 
    static public final String T8000 = "";
}

Compilation using default heap settings just produce Out of memory exception but once JVM granted plenty of ram (-Xmx32g) it ends up with exception:

java.lang.OutOfMemoryError: Required array length too large

RoboVM went beyond of JVM limits trying to allocate byte array with size longer than JVM supported (Integer.MAX_VALUE - 2 gig). In other words – giving all memory you have to JVM heap will not solve this issue.

Root case

OOM happens when RoboVM compiles java code into LLVM IR code. This constant string only class doesn’t fit in MAX possible 2 GB java string. Let’s see what is in there.

tutorial: RoboVM and dependent watchOS app

|

tutorial: marshaling NSArray of protocols

|

Backtrack: issue, gitter

It becomes problematic to marshal methods that return NSArray<id<PROTOCOL>> like bellow:

- (void)readerSession:(NFCReaderSession *)session 
        didDetectTags:(NSArray<__kindof id<NFCTag>> *)tags;

Issue 1: NSArray

Direct binding will not compile:

fixed: idea debug session hang + launcher classes rework

|

Issue

There was a rare bug when debug session in Intellij Idea failed and hang. It was not possible to terminate it without reopening the project.

Root case

Issue was reproduced while by having broken deployment to simulator step that allowed to investigate and locate the root case.
The root case – Idea hangs on waiting for stdout/stderr to be closed on terminated (due fail to deploy to sim) process.
Root case 1: the way stdout was redirected:

stdout -> FIFO FILE -> input stream

Root case 2: wrapper process class ProcessProxy around launcher process that was never returned stream to Idea but was never closing it (here it hangs).

The fix: launcher classes rework

  • Currently, there are three launchers: ios simulator, ios device, console. All use own launcher. The fix introduced common functionality as separated AbstractLauncherProcess class. Specific implementations just extends it;
  • FIFO and redirect logic massively simplified. No FIFO, just simple Piped streams. Added observable stream to allow a plugin to monitor stdout (as part of FIFO replacement) - required by JUNIT plugin;

Console target fixes

  • fixed not working debugging of console target;
  • added support for STDIN pipe to console target;
  • console template fixed to include required <target>console</target>.

Code

PR481

AltPods: pods updated - 1.7.0-SNAPSHOT

|

AltPods were update to v1.7.0-SNAPSHOTS to sync with recent releases. Part of list didn’t receive any API update hovewer bindings were re-generated against recent version of frameworks. Update list look as bellow:

New framework

Updated pods

Unchanged pods

These pods were pushed to https://oss.sonatype.org/content/repositories/snapshots maven repo under 1.7.0-SNAPSHOTS version. Source code @github

Updates are not fullty tested, please open issue if bug found.

NB: AltPods – are robopods kept at my personal standalone repo. This is done to prevent main robopods repo from turning into code graveyard. As these pods have low interest of community and low chances to be updated.

Debugger: maintenance 2020.4

|

PR475 delivers an amount of fixes related to the debugger:

fixed: debugger crashes when setting breakpoints in specific method

Root case is changes in previous improvement: debugger reports now only line number where breakpoint can be hit.
Improvement was to re-use breakpoint map structure as lines available for breakpoints. Debugger resolves that map when line map is being requested.
If all lines in the method are available for breakpoints and number of lines is even to eight this results in available for breakpoints map completely filled with zeroes. Linker puts zero filled memory structures to __bss data segment. Debugger was not expecting symbols in this segment and was crashing with unable to resolve symbol exception.

The fix: refactored buffer reader and added ZeroReader to support .bss data segment.

fixed: broken stepping over line that loads inner class

this happens when stepping over code similar to bellow:

inner class Test2()
fun test() {
    print(1)
    Test2()    // stepping over 
    print(2)   // will not stop here first time     
}

Root case: is class loaded event from device/simulator that suspends thread. Once thread is suspended all stepping constraints removed. Debugger after receiving class loaded even tells the target to resume. But there is no stepping constraint available anymore.
The fix: restore stepping constraint in device on thread resume if stepping is active.

fixed: stepping inside kotlin lambda

The issue can be observed in code similar to bellow:

      fun test() {
/*7*/    list.forEachIndexed { index, str ->
/*8*/       text = str + index       // <-- will not step or stop at this line 
/*9*/    }
     }

Kotlin injects collections code to user class under extended line number range and uses SMAP to map extended range into class line number range. Example above turned into following pseudo code after SMAP un-mapping:

    fun test() {
/*7*/   val l = list as Iterable<String>; var idx = 0;
/*8*/   val it = l.iterator(); while(it.hasNext()) { val str = it.next(); text = str + idx++ }
    }

As result there is multiple operation an line 8. but only one breakpoint/stop point will be instrumented before first operation in line. This results in breakpoint/stop point injected before l.iterator() and no one inside the loop.

The fix: inject stop/breakpoint hooks before SMAP line numbers unmapped and code get collapsed in single line.

optimizations: application binary stripped

Binary was not stripped before as strip removes local symbols as result debugger was not able to resolve required symbols. To support stripping its requires to pre-save symbols debugger depends on. This was done by putting these into exported symbols list.
Having binary stripped results in following (sample data based on empty project, x86_64 simualator):

  • application binary size reduced: from 113M -> 45M, faster deployment;
  • number of symbols debugger have to parse reduced: from 317K -> 28K, faster parsing(10x time) and debugger startup, smaller memory footprint as no need to keep never used ~300K symbols (about 60M);

rework: refactored buffer reader

Code to manipulate memory buffer refactored to provide abstract interface instead of ByteBuffer centric implementation. Code refactored to use interfaces that allows to provide implementation with different data source. For example NullReader to simulate reading from .bss segment. Also, it allows to have future improvements required for windows builds (workaround for mapped file issues).

rework: .spfpoffset value fetched during compilation (was runtime)

Initially it was mistake to make .spfpoffset symbols global as it resulted in many “symbol already defines” error.
Root case of it is LLVM emitter who adds .spfpoffset for all functions (even always inline or private). As result there was many private symbols with same name turned into global.
Situation was fixed (for having binary stip sake) by reworking the way .spfpoffset retrieved:
was:

  • symbols were referenced in llvm.used just to survive dead code elimination;
  • debugger resolved .spfpoffset runtime and fetched value;

now:

  • during compilation .spfpoffset is being fetched from object file;
  • value added to class debug information;
  • .spfpoffset is not required anymore and can be eliminated as not used.

other changes: dependency order changed

Debugger module reworked to be independed from LLVM/Compiler module. Now its stand-alone module. Instead following was changed:

  • compiler module has direct dependency to debugger;
  • idea/eclipse/maven/gradle plugins don’t depend anymore on debugger (as it is part of compiler now);

Support for swift dependencies for static libs/frameworks

|

RoboVm compiler is able to detect swift usage in dynamic frameworks and copies all required swift libraries into application bundle.
Same logic (calling otool -L) doesn’t work for static libraries (.a) and static frameworks.
While xcframework and better logic (instead of otool -L) is still in development workaround was introduced to unblock community:

swift libraries have to be manually listed in robovm.xml:

Example:

<libs>
    <lib>libswiftCore.dylib</lib>
    <lib>libswiftCore.dylib</lib>
    <lib>libswiftCoreFoundation.dylib</lib>
    <lib>libswiftCoreGraphics.dylib</lib>
    <lib>libswiftCoreImage.dylib</lib>
    <lib>libswiftDarwin.dylib</lib>
    <lib>libswiftDispatch.dylib</lib>
    <lib>libswiftFoundation.dylib</lib>
    <lib>libswiftMetal.dylib</lib>
    <lib>libswiftObjectiveC.dylib</lib>
    <lib>libswiftQuartzCore.dylib</lib>
    <lib>libswiftUIKit.dylib</lib>
</libs>

Simplest way to find out list of required swift libraries is to use otool -l libOne.a | grep "\-lswift".

Now compiler detects swift library in dependencies and applies logic similar to dynamic frameworks:

  • adds swift library path to library search path;
  • link binary with these libraries;
  • copies swift libraries into application bundle;

Code was delivered as PR474.

AltPods: pods updated - 1.6.0-SNAPSHOT

|

AltPods were update to v1.6.0-SNAPSHOTS to sync with recent releases. Part of list didn’t receive any API update hovewer bindings were re-generated against recent version of frameworks. Update list look as bellow:

New framework

Updated pods

Unchanged pods

These pods were pushed to https://oss.sonatype.org/content/repositories/snapshots maven repo under 1.6.0-SNAPSHOTS version. Source code @github

Updates are not fullty tested, please open issue if bug found.

NB: AltPods – are robopods kept at my personal standalone repo. This is done to prevent main robopods repo from turning into code graveyard. As these pods have low interest of community and low chances to be updated.

iOS 13.4 bindings preview (beta2)

|

iOS 13.4

bindings have arrived as PR458.
selectively updated frameworks based on api diff
binding preview based on Xcode11.4 BETA 2, following was updated:

  • AuthenticationServices
  • AutomaticAssessmentConfiguration (Added)
  • AVFoundation
  • CallKit
  • CarPlay
  • ClassKit
  • Contacts
  • CoreBluetooth
  • CoreGraphics
  • CoreLocation
  • CoreMedia
  • CoreText
  • GameKit
  • ImageCaptureCore
  • Intents
  • LocalAuthentication
  • MetalPerformanceShaders (also added missing classes from previous iOS)
  • NetworkExtension
  • PassKit
  • StoreKit
  • UIKit
  • WebKit

Framework improved -- exposing ObjectiveC classes to natively create instances

|

Previous improvement simplified way Framework target was created right out the box without need to write native code. But it didn’t allow creating objects native way and workaround was to use single point singleton and consider objects as protocols. RoboVM creates objects runtime using objc/runtime API so there is no information available during linkage phase. And a try to use it will cause symbol not found exception like this:

Undefined symbols for architecture x8664: “_OBJC_CLASS$_RoboClass”, referenced from: objc-class-ref in ViewController.o

The goal of this Improvement:

  • hide any need to initialize JVM from user;
  • allow natively creating and using of java classes as native.

In general it should be enough:

  • drop framework in Xcode project;
  • instantiate and use classes like bellow:
    let demo = MyFrameworkDemo(text: "demo")!
    print(demo.roboVmVersion()!)
    // or
    MyFrameworkDemo *demo = [[MyFrameworkDemo alloc] initWithText:@"demo"];
    NSLog(@"%@", [demo roboVmVersion]);
    

(skip long read and start creating framework)

Step one. OBJC_CLASS_$_${CLASSNAME} structure