Few cents about my commits

iOS 13.4 bindings preview (beta2)

|

iOS 13.4

bindings have arrived as PR458.
selectively updated frameworks based on api diff
binding preview based on Xcode11.4 BETA 2, following was updated:

  • AuthenticationServices
  • AutomaticAssessmentConfiguration (Added)
  • AVFoundation
  • CallKit
  • CarPlay
  • ClassKit
  • Contacts
  • CoreBluetooth
  • CoreGraphics
  • CoreLocation
  • CoreMedia
  • CoreText
  • GameKit
  • ImageCaptureCore
  • Intents
  • LocalAuthentication
  • MetalPerformanceShaders (also added missing classes from previous iOS)
  • NetworkExtension
  • PassKit
  • StoreKit
  • UIKit
  • WebKit

Framework improved -- exposing ObjectiveC classes to natively create instances

|

Previous improvement simplified way Framework target was created right out the box without need to write native code. But it didn’t allow creating objects native way and workaround was to use single point singleton and consider objects as protocols. RoboVM creates objects runtime using objc/runtime API so there is no information available during linkage phase. And a try to use it will cause symbol not found exception like this:

Undefined symbols for architecture x8664: “_OBJC_CLASS$_RoboClass”, referenced from: objc-class-ref in ViewController.o

The goal of this Improvement:

  • hide any need to initialize JVM from user;
  • allow natively creating and using of java classes as native.

In general it should be enough:

  • drop framework in Xcode project;
  • instantiate and use classes like bellow:
    let demo = MyFrameworkDemo(text: "demo")!
    print(demo.roboVmVersion()!)
    // or
    MyFrameworkDemo *demo = [[MyFrameworkDemo alloc] initWithText:@"demo"];
    NSLog(@"%@", [demo roboVmVersion]);
    

(skip long read and start creating framework)

Step one. OBJC_CLASS_$_${CLASSNAME} structure

Idea plugin: run configuration maintenance

|

Why

Idea run configuration might become invalid if configured items become invalid, like:

  • simulator is not available in next xcode;
  • signing identity or provision profile expired or removed;

Configuration editor was hiding errors

In this case compilation failed with runtime exception. But once run configuration editor is opened instead of pointing to invalid items editor was auto picking up first one available. As result user was not able to see a problem and to have configuration overridden changes to it to be done. Otherwise apply button was not active.

Name instead unique id was used as identifier

To identify signing identity/simulator/provisioning profile its name was saved. For auto value just auto text placeholder was saved and checked against. This is subject for name collisions.

Whats changed

Auto mode explicitly specified

Field that might be auto (Signing identity/Provisioning profile/Simulator) are now have separate parameter in config file that specifies entry type: auto or explicit ID. And auto mode is identified by it and not string constant in entry name.

Explicit identifier instead of name

When explicit entry is used (e.g. not auto) entries id is saved instead of entry name (udid for profile/simulator, footprint for signing identity). No collision anymore.

Auto mode for simulator

Simulator field receives two modes auto iPhone and auto iPad. These modes acts similarly to gradle’s one ‘launchIPhoneSimulator’/launchIPadSimulator. Also in case there is no simulator available in system it will allows to recover broken run configuration.

Quick fix for broken fields

Problem fields can be quick fixed to auto value with Fix button.

arm64 is default for device target

Default thumbv7 was a problem as is not available since ios11 (on most devices today). This caused deployment error at very end of run cycle. Having arm64 default improves user experience.

no more x86 (32bit) option for ios11+ simulators

Similar issue to arm64. It 32bit code doesn’t run on ios11+ simulators.

no more don't sign options

don't sign option was for usage with jailbroken ios devices. While having free provisioning from Apple this option have no more sense. Also ldid binary was removed. If don't sign option is specified from gradle/maven ADHOC signing will be performed.

bonus1: gradle import of template projects was fixed

Created from template gradle projects were broken due:

  • gradle was set to use RoboVM SDK (and it is not operational any more for tools);
  • Java source code level was set to Java12+;

Now: gradle uses internal JDK, java source level forced to Java8.

bonus2: added support for Apple Development certificates

These were not part of regex and were ignored when auto identity was specified.

bonus3: suppressed -559038737 error code

If simulator process is terminated by Idea strange error message was printed into console log:

Process finished with exit code -559038737

This code is UNKNOWN_CODE constant in Executor. Now it is being recognized and replaced with 0.

Code

Code was delivered as PR456

AltPods: pods updated - 1.5.0-SNAPSHOT

|

AltPods were update to v1.5.0-SNAPSHOTS to sync with recent releases. Part of list didn’t receive any API update hovewer bindings were re-generated against recent version of frameworks. Update list look as bellow:

Updated pods

Unchanged pods

These pods were pushed to https://oss.sonatype.org/content/repositories/snapshots maven repo under 1.5.0-SNAPSHOTS version. Source code @github

Updates are not fullty tested, please open issue if bug found.

NB: AltPods – are robopods kept at my personal standalone repo. This is done to prevent main robopods repo from turning into code graveyard. As these pods have low interest of community and low chances to be updated.

Debugger: maintenance 2020

|

PR455 delivers amount of fixes related to the debugger:

fix: breakpoint were not hit if stepping over single line cycle

Code like bellow once stepped over were not breaking anymore.

   while(true)
      foo(); // breakpoint here

Root case: stepping event was checked before breakpoint one and stepping was reported to JDPW client. Idea skipped this event as line number wasn’t changed.

fixed: debugger crash resolving Thread objects that wasn’t started

Root case: such objects have no native thread attached yet so it was crashing on NPE due missing native thread pointer.

improvement: debugger reports now only line number where breakpoint can be hit

This enables IDE to display invalid breakpoints as on image bellow:

To support this special bit map is generated that specifies if line is valid. As storage bptable is used. This structure is used by injected hook code to check if there is breakpoint set at specific line. The trick with this enhancement that this structure has bit set for lines where no hook injected. As result no breakpoint will be triggered by these bits (as there is no corresponding hook code injected) and debugger is know lines without hooks.

improvement: Idea settings: debugger/stepping filter – extended with dalvik.*, libcore.*

This is done to disable stepping into these class paths similar how stepping into Runtime classes is disabled. As it often stops there when stepping in user code. This can be disable in Idea preferences:

improvement: removed synthetic bro-bridges/bro-callbacks from back-stack entries

These autogenerated and not useful as provides no useful information:

fixed: not working step-out

In case stepping out in method/class that is ignored (like Runtime classes are ignored in Idea stepping settings) debugger should pick up valid not ignored back stack entry. Otherwise step out wasn’t working.

Why JavaFXPorts is crashing on ios13.3

|

There was a report on Gitter channel saying that there is an issue on ios13.3 in JavaFX port:

@javasunsFX_twitter Anyone has an idea what has changed on iOS 13.3 and JavaFX crashes when calling Stage.show(), which in turn calls “IosWindow -> GlassWindow.m -> _createWindow”?

Issue was confirmed and investigated to check if RoboVM is affected. Crash log looks like bellow:

Thread 0 name:  Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0   libsystem_kernel.dylib        	0x00000001b4f4c930 __abort_with_payload + 8
1   libsystem_kernel.dylib        	0x00000001b4f50f24 abort_with_payload_wrapper_internal + 100
2   libsystem_kernel.dylib        	0x00000001b4f50ec0 abort_with_payload_wrapper_internal + 0
3   libsystem_pthread.dylib       	0x00000001b4e75ed0 DYLD-STUB$$OUTLINED_FUNCTION_0 + 0
4   libsystem_pthread.dylib       	0x00000001b4e750a8 pthread_join + 0
5   Foundation                    	0x00000001b54089d4 +[NSThread currentThread] + 96
6   Watchdog                      	0x00000001049b0858 Java_com_sun_glass_ui_ios_IosWindow__1createWindow + 392
7   Watchdog                      	0x0000000104c69bc0 [J]com.sun.glass.ui.ios.IosWindow._createWindow+ 4856768 (JJI)J + 64

And abort payload is pthread_t was corrupted. Memory corruption is hard to investigate as cause of it is not same place (and time) where it happened. To investigate issue JavaFX was build as debug and it worked. Future investigation shown that it fails if compiled with NDEBUG define. As JavaFX is huge project it is hard to analyze all code. Quickest way to find a bug in unknown code is to slice it and isolate failing code to few lines.

Root case

JavaFX source was sliced and ripped to few lines that showed the root case:

  • GLASS_POOL_PUSH macro causes the issue and corrupts the memory when writing to _GlassThreadData;
  • _GlassThreadData is never allocated because (GlassThreadData*)pthread_getspecific(GlassThreadDataKey) returns data;
  • return value of pthread_getspecific is undefined in case of invalid key (and it is!);
  • and key is undefined:

And the root case of issue why key is undefined is the code how its initialized, GlassApplication.m:635:

    assert(pthread_key_create(&GlassThreadDataKey, NULL) == 0);

When NDEBUG is defined assets are not being compiled and pthread_key_create(&GlassThreadDataKey, NULL) is never executed. It was working on pre ios13.3 just due lucky case as it was corrupting not-critical data.

The fix: is to get pthread_key_create out of assert.

Bottom line

RoboVM compiler is not affected!

RoboVM 2.3.10-SNAPSHOT, experimental build

|

@Tomski made first 2.3.10-SNAPSHOT build today. It artifacs deployed to sonatatype. Idea snapshot plugin is available for download.

PLEASE NOTE:

  • IMPORTANT: Idea plugin is now a .zip file and Safari will unpack it (by default) which will make it not functional. It has to be downloaded as .zip file and installed using .zip file, not with .jar file inside zip/unpacked folder.
  • At moment of writing latest release is 2.3.8 and latest snapshot is 2.3.9-SNAPSHOT.
  • 2.3.10-SNAPSHOT is experimental as contains many introduced changes in compier/api/bindings which were not deeply tested. While 2.3.9-SNAPSHOT version is being kept for possible hotfixes.
  • 2.3.10-SNAPSHOT is build from jdk12 branch and it names might be confusing. It doesn’t deliver JDK12 functionality to RoboVM but was opened to allow RoboVM compiler be running on JDK9+. As result it hosts 2.3.10 version.

What’s changed

Source tree: migrating to jdk12

  • maven project structure reformated;
  • removed sonatype root parent pom;
  • improved dependency management (moved to root pom);
  • plugin versions/dependencies updated to recent where possible;
  • unit tests fixed to run on jdk9+/openjdk.

Bindings

Compiler changes (fixes and new tricks)

  • Generic class arguments and @Block parameters: PostFix #1, pr419
  • Support for non-static @Bridge method in enums classes: PostFix #2, pr420
  • Support for @Block member in structs: PostFix #3, pr421
  • [fixed]Compilation failed on @Bridge annotate covariant return synthetic method: PostFix #4, pr422
  • Support for Struct.offsetOf in structs: PostFix #5, pr431
  • workaround for missing objc classes(ObjCClassNotFoundException), PostFix #8, pr442
  • Hooray! experimental and formal bitcode support, PostFix #9, pr443
  • [fix] error code -34018 when using Security API on simulator: pr447, post

Idea plugin

  • fixed annoying Android gradle faced #242, post;
  • plugin is now built with gradle script, see migrating to jdk12;
  • code was cleaned up, upgraded to Java8 code level, and deprecated API usage were removed where possible to support Idea 2019.3, “no Xcode dialog” is not blocking anymore: pr434;
  • disabled generate separate IDEA module per source set: pr449, post;

Happy coding! Please report any issue to tracker.

Idea: disabling `generate separate IDEA module per source set`

|

Idea 2019.2 deprecated generate separate IDEA module per source set option as mentioned in their blog post, reworked Gradle settings dialog. For RoboVM this option is important as it is required to be disabled. Having it enabled it will produce three modules once gradle file is imported:

module (content root: ./ )
    module-main (content root: ./src/main)
    module-test (content root: ./src/test)

Root case

Its a problem for Idea plugin to find out the module eligible to run. As one of criteria is presence of robovm.xml in content root of module. Example above shows that module with code is module-main but its content root points to source folder not module root folder itself (and there is no robovm.xml). Group module module contains valid content root with robovm-xml but no source root. As result run/debug dialog shows no valid module to run/debug.

Workaround

Simulator: enabling Security framework(fix for error -34018)

|

@Tom-ski reported that Security api like SecItemAdd fails in RoboVM applications while running on simulator with error -34018:

The operation couldn’t be completed. (OSStatus error -34018 - Client has neither application-identifier nor keychain-access-groups entitlements)

Same time there is no issue for application when running on device. There are multiple reports about similar issues on web, and one on Stackoverflow contain reference to statement from Xcode8.1 release notes:

Keychain APIs may fail to work in the Simulator if your entitlements file doesn’t contain a value for the application-identifier entitlement.

Quick test reproduced this issue with Java code like this:

SecAttributes attributes = new SecAttributes();
attributes.set(SecQuery.Keys.Class(), SecClass.Values.GenericPassword());
attributes.setAccount("My key");
attributes.set(SecValue.Keys.Data(), new CFString("My secretive bee 🐝"));
attributes.setAccessible(SecAttrAccessible.WhenUnlocked);
try {
    CFType res = SecItem.add(attributes);
    System.out.println(res);
} catch (OSStatusException e) {
    e.printStackTrace();
}

But there is an error possible in bindings, to check this native library with simple class was generated and called from RoboVM project:

@implementation SimpleTest
-(void) test {
    NSMutableDictionary *queryAdd = [NSMutableDictionary dictionary];
    [queryAdd setValue:kSecClassGenericPassword forKey:kSecClass];
    [queryAdd setValue:itemKey forKey:kSecAttrAccount];
    [queryAdd setValue:[itemValue dataUsingEncoding:NSASCIIStringEncoding] forKey:kSecValueData];
    [queryAdd setValue:kSecAttrAccessibleWhenUnlocked forKey:kSecAttrAccessible];
    int resultCode = SecItemAdd(queryAdd, nil);
    NSLog(@"SecItemAdd %d", resultCode);
}
@end

And result was same error -34018.

the fix

Same time this ObjC code successfully runs on simulator from Xcode. That means there is something in build/signing process missing in case of RoboVM. Looking throw Report navigator shows following:

  • Xcode signs simulator binary with ADHOC identity. But it is not enough;
  • Xcode adds entitlements during signing. But signing with simple get-task-allow (as mentioned in web) crashes App on start with Invalid entitlements exception.
  • It turns out that Xcode uses com.apple.security.get-task-allow instead. With it app doesn’t crash but Security API still doesn’t work;
  • Inspecting binary built by Xcode for application-identifier discloses another entitlements (besides one in signature) stored in __TEXT __entitlements and it contains following:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>application-identifier</key>
	<string>6THE3YW9HQ.bundle.id</string>
	<key>keychain-access-groups</key>
	<array>
		<string>6THE3YW9HQ.bundle.id</string>
	</array>
</dict>
</plist>

Having such section embedded into RoboVM application solves the issue. Success !

Bottom line

As result:

  • signing of binary built for simulator is not required, but was added to be compatible with what Xcode is doing;
  • RoboVM compiler now embeds __entitlement section similar to Xcode (but this is not documented anywhere).

Code was delivered as PR447

iOS13 PostFix #10: glkit -- missing functions (static inline now)

|

Report in gitter shown another problem in bindings: java.lang.UnsatisfiedLinkError: Optional @Bridge method GLKMatrix4.create(FloatBuffer;)GLKMatrix4; not bound

Quick investigation shown that this method is not available runtime anymore as tons of functions converted into static inline. Today this functions looks as bellow:

GLK_INLINE GLKMatrix4 GLKMatrix4MakeWithArray(float values[16])
{
    GLKMatrix4 m = { values[0], values[1], values[2], values[3],
                     values[4], values[5], values[6], values[7],
                     values[8], values[9], values[10], values[11],
                     values[12], values[13], values[14], values[15] };
    return m;
}

As result RoboVM is not able to resolve it runtime and use it. As long as it is not available runtime there are two options to use it:

  • get all API and .c file, build it into object file and distribute as part of bindings;
  • implement missing API in java (porting it).

While first option is preferable due performance concerns second approach will be implemented as quickest solutions. All changes were added to ios13.2 bindings branch as commit.

Other postfixes: