Full microservice stack generation using import-jdl #8335

deepu105 · 2018-09-15T19:03:40Z

TODO:

Rebuild client when only entities are created
Move tests to CLI
Add test for multi app genefation
Please make sure the below checklist is followed for Pull Requests.
Travis tests are green
Tests are added where necessary
Documentation is added/updated where necessary
Coding Rules & Commit Guidelines as per our CONTRIBUTING.md document are followed

deepu105 · 2018-09-15T19:06:52Z

@MathieuAA @agoncal would you like to test drive?

create the jdl with multiple apps in a folder, for example app.jdl

application {
  config {
    baseName myFirstApp,
    applicationType gateway,
    packageName com.mycompany.myfirstapp,
  }
  entities * except C, D
}

application {
  config {
    baseName mySecondApp,
    applicationType microservice,
    serverPort 8091,
  }
  entities E
}

application {
  config {
    baseName myThirdApp,
    applicationType microservice,
    serverPort 8092,
  }
  entities F
}

entity A
entity B
entity C
entity D
entity E
entity F

microservice E with mySecondApp

Run jhipster import-jdl app.jdl in that folder and see the magic :)

deepu105 · 2018-09-15T19:07:52Z

@jhipster/developers more testing is welcome as well. Now we can create a full microservice stack under 5 minutes. this is gonna be awesome for some demo :)

deepu105 · 2018-09-15T19:08:23Z

also all apps are created in parallel 😉

PierreBesson · 2018-09-15T19:38:01Z

It's decided I'm using this in my demo in two weeks.

MathieuAA · 2018-09-15T19:41:37Z

@deepu105 by all means, it's a good test file but the entity exclusions might confuse users if they try to understand during a presentation (<- @PierreBesson, you may have to simplify it a bit so that anyone can read it quickly)

deepu105 · 2018-09-15T19:53:14Z

@MathieuAA ofcourse this is only for tests, I'll be using a more realistic use case (my e-commerce app from the book)

deepu105 · 2018-09-15T19:54:32Z

@MathieuAA btw is this valid as well?

application {
  config {
    baseName myFirstApp,
    applicationType monolith,
    packageName com.mycompany.myfirstapp,
  }
  entity A
entity B
entity C
entity D
}

MathieuAA · 2018-09-15T19:58:04Z

@deepu105 declare them outside of the application scope and just "import" them using entities <YOUR_ENTITIES>. The reason it's this way is to:

make it usable using multiple files (for instance, one for entities, others for applications),
not be dependent of an "import" keyword
not repeat the entity declaration step if you want to reuse entities

deepu105 · 2018-09-15T19:58:36Z

ok no probs, just checking

MathieuAA · 2018-09-15T20:34:02Z

generators/upgrade/index.js

            assertJHipsterProject() {
                const done = this.async();
-                if (!this.getJhipsterAppConfig()) {
+                if (!this.getAllJhipsterConfig().baseName) {


I suggest moving the call to this.getAllJhipsterConfig() to a separate variable as the method does some I/O and someone may be tempted to just reuse a similar call as this.getAllJhipsterConfig().baseName without assigning the result of this.getAllJhipsterConfig() to a variable (and making the I/O calls n times)

mraible · 2018-09-15T23:24:38Z

Does JDL-Studio need to be updated to support this syntax? If I try to save my full apps.jh, it fails with the following error.

Expected "/*", "angularSuffix", "dto", "entity", "enum", "filter", "microservice", 
"noFluentMethod", "paginate", "relationship", "search", "service", "skipClient", 
"skipServer", ['\n'|'\t'|'\r'|' '| | ], [A-Z_], or end of input but "a" found. -> line: 1

mraible

This is awesome! However, I did notice that you have to specify serviceDiscoveryType eureka for the gateway. Should this be the default?

mraible · 2018-09-15T23:31:34Z

cli/import-jdl.js

+    generator, application, withEntities, inAppFolder
+}) => {
+    const baseName = application[GENERATOR_NAME].baseName;
+    logger.info(`Generating application ${baseName} in a new parellel process`);


parellel > parallel

mraible · 2018-09-15T23:31:44Z

cli/import-jdl.js

+};
+
+const generateEntity = (baseName, inAppFolder, generator, entity) => {
+    logger.info(`Generating entities for application ${baseName} in a new parellel process`);


parellel > parallel

mraible · 2018-09-15T23:44:22Z

I tried the following JDL and it fails to compile when I try to start the store app.

application {
  config {
    baseName gateway,
    applicationType gateway,
    authenticationType oauth2,
    prodDatabaseType postgresql,
    serviceDiscoveryType eureka,
    testFrameworks [protractor]
    packageName com.okta.developer.gateway
  }
  entities Blog, Post, Tag, Product
}

application {
  config {
    baseName blog,
    applicationType microservice,
    authenticationType oauth2,
    prodDatabaseType postgresql,
    searchEngine elasticsearch,
    serverPort 8081,
    packageName com.okta.developer.blog
  }
  entities Blog, Post, Tag
}

application {
  config {
    baseName store,
    applicationType microservice,
    authenticationType oauth2,
    databaseType mongodb
    searchEngine elasticsearch,
    serverPort 8082,
    packageName com.okta.developer.store
  }
  entities Product
}

entity Blog {
  name String required minlength(3),
  handle String required minlength(2)
}

entity Post {
  title String required,
  content TextBlob required,
  date Instant required
}

entity Tag {
  name String required minlength(2)
}

relationship ManyToOne {
  Blog{user(login)} to User,
  Post{blog(name)} to Blog
}

relationship ManyToMany {
  Post{tag(name)} to Tag{post}
}

paginate Post, Tag with infinite-scroll

entity Product {
  title String required,
  price BigDecimal required,
  image ImageBlob required
}

Error is

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project store: Compilation failure: Compilation failure:
[ERROR] /Users/mraible/apps-8185/store/src/main/java/com/okta/developer/store/domain/Product.java:[3,33] package org.hibernate.annotations does not exist
[ERROR] /Users/mraible/apps-8185/store/src/main/java/com/okta/developer/store/domain/Product.java:[4,33] package org.hibernate.annotations does not exist

Product.java looks as follows

package com.okta.developer.store.domain;

import org.hibernate.annotations.Cache;
import org.hibernate.annotations.CacheConcurrencyStrategy;
import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Field;
import org.springframework.data.mongodb.core.mapping.Document;
import javax.validation.constraints.*;

import java.io.Serializable;
import java.math.BigDecimal;
import java.util.Objects;

/**
 * A Product.
 */
@Document(collection = "product")
@org.springframework.data.elasticsearch.annotations.Document(indexName = "product")
public class Product implements Serializable {

    private static final long serialVersionUID = 1L;

    @Id
    private String id;

    @NotNull
    @Field("title")
    private String title;

    @NotNull
    @Field("price")
    private BigDecimal price;


    @Field("image")
    private byte[] image;

    @Field("image_content_type")
    private String imageContentType;

    // jhipster-needle-entity-add-field - JHipster will add fields here, do not remove
    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getTitle() {
        return title;
    }

    public Product title(String title) {
        this.title = title;
        return this;
    }

    public void setTitle(String title) {
        this.title = title;
    }

    public BigDecimal getPrice() {
        return price;
    }

    public Product price(BigDecimal price) {
        this.price = price;
        return this;
    }

    public void setPrice(BigDecimal price) {
        this.price = price;
    }

    public byte[] getImage() {
        return image;
    }

    public Product image(byte[] image) {
        this.image = image;
        return this;
    }

    public void setImage(byte[] image) {
        this.image = image;
    }

    public String getImageContentType() {
        return imageContentType;
    }

    public Product imageContentType(String imageContentType) {
        this.imageContentType = imageContentType;
        return this;
    }

    public void setImageContentType(String imageContentType) {
        this.imageContentType = imageContentType;
    }
    // jhipster-needle-entity-add-getters-setters - JHipster will add getters and setters here, do not remove

    @Override
    public boolean equals(Object o) {
        if (this == o) {
            return true;
        }
        if (o == null || getClass() != o.getClass()) {
            return false;
        }
        Product product = (Product) o;
        if (product.getId() == null || getId() == null) {
            return false;
        }
        return Objects.equals(getId(), product.getId());
    }

    @Override
    public int hashCode() {
        return Objects.hashCode(getId());
    }

    @Override
    public String toString() {
        return "Product{" +
            "id=" + getId() +
            ", title='" + getTitle() + "'" +
            ", price=" + getPrice() +
            ", image='" + getImage() + "'" +
            ", imageContentType='" + getImageContentType() + "'" +
            "}";
    }
}

Am I doing something wrong?

The .yo-rc.json for the store app looks as follows

{
  "generator-jhipster": {
    "applicationType": "microservice",
    "clientPackageManager": "npm",
    "databaseType": "mongodb",
    "devDatabaseType": "h2Disk",
    "enableHibernateCache": true,
    "enableSwaggerCodegen": false,
    "enableTranslation": true,
    "jhiPrefix": "jhi",
    "languages": [
      "en",
      "fr"
    ],
    "messageBroker": false,
    "nativeLanguage": "en",
    "packageName": "com.okta.developer.store",
    "packageFolder": "com/okta/developer/store",
    "prodDatabaseType": "mysql",
    "searchEngine": "elasticsearch",
    "serviceDiscoveryType": false,
    "skipClient": true,
    "skipServer": false,
    "testFrameworks": [],
    "useSass": false,
    "websocket": false,
    "baseName": "store",
    "authenticationType": "oauth2",
    "serverPort": "8082",
    "jhipsterVersion": "5.3.1",
    "buildTool": "maven",
    "cacheProvider": "hazelcast",
    "clientFramework": "angularX",
    "skipUserManagement": true
  },
  "entities": [
    "Product"
  ]
}

mraible · 2018-09-16T00:16:38Z

I changed my store app to use the following JDL and now it compiles:

application {
  config {
    baseName store,
    applicationType microservice,
    authenticationType oauth2,
    databaseType mongodb,
    devDatabaseType mongodb,
    prodDatabaseType mongodb,
    enableHibernateCache false,
    cacheProvider hazelcast,
    searchEngine elasticsearch,
    serverPort 8082,
    packageName com.okta.developer.store
  }
  entities Product
}

Not only that, but all Protractor tests pass. This is a sweet feature! 🎉

mraible · 2018-09-16T01:34:17Z

One issue I found. I'm not sure if it relates to this feature or import-jdl in general. If I don't specify searchEngine elasticsearch on my gateway, the UI is generated without a search field.

It seems like this should be read from the microservice apps (which have elasticsearch specified) rather than requiring it on the gateway.

deepu105 · 2018-09-16T07:34:20Z

Thanks for the feedback guys. I'll see what are the issues that can be handled in the subgen and the issues that needs to fixed in JHipster core. @MatheueAA I'll create a new issue in JH core and list down the issues found is that ok with you?

…

On Sun, 16 Sep 2018, 3:34 am Matt Raible, ***@***.***> wrote: One issue I found. I'm not sure if it relates to this feature or import-jdl in generator. If I don't specify searchEngine elasticsearch on my gateway, the UI is generated without a search field. It seems like this should be reach from the microservice apps (which have this specified) rather than requiring it on the gateway. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8335 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABDlF8iqcFtvw7MQFLJjAF4VRWr5Gc-uks5ubaqdgaJpZM4Wqft8> .

MathieuAA · 2018-09-16T07:57:37Z

Please do, I have time to fix them today

deepu105 · 2018-09-16T10:38:34Z

@pascalgrimaud might need your expertise here. I was thinking of setting up a Travis test with a full stack.
This is what I'm thinking

We can have a new script which runs import jdl to generate a full microservice stack
In one of the matrix we specify a param and we can use this script instead of the regular one
If possible start everything with docker and run the protractor tests on gateway

WDYT?

deepu105 · 2018-09-16T10:52:49Z

@mraible JDL studio needs update to work with this. I'll see what can be done

pascalgrimaud · 2018-09-16T11:01:39Z

@deepu105 : yes it's doable and it will improve a lot our tests for microservices !

typos corrected

deepu105 · 2018-09-16T16:07:30Z

@mraible I have made the JDL studio to work with this, for now, the old studio here works (with autocompletion and syntax highlight) https://www.jhipster.tech/jdl-studio/ Its more of a hack for now, as @MathieuAA suggested it needs a full rewrite in order to properly utilize chevrotain and its autosuggestion, lint capabilities

For the new studio in JHipster online probably will have to wait for a release by @jdubois

deepu105 · 2018-09-16T16:14:06Z

@MathieuAA this PR is ready to merge, but i'll wait for your updates on JH core

MathieuAA · 2018-09-16T16:28:19Z

@deepu105 Okay! As usual it's awesome work. Concerning the changes, I'm refactoring the code (the JDL app part). I'll try to be done by today so that this can be merged.

deepu105 · 2018-09-19T08:11:45Z

My first bounty claim https://opencollective.com/generator-jhipster/expenses/4982
I feel naughty 😉

jdubois · 2018-09-19T08:39:14Z

@deepu105 lucky you, I still haven't claimed one :-(

deepu105 · 2018-09-19T10:02:13Z

You should it feels good :) Thanks & Regards, Deepu

…

On Wed, Sep 19, 2018 at 10:39 AM Julien Dubois ***@***.***> wrote: @deepu105 <https://github.com/deepu105> lucky you, I still haven't claimed one :-( — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8335 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABDlF3ueCOqcs4FEkbuEjAcY026ovUwBks5ucgK6gaJpZM4Wqft8> .

jsm174 · 2018-09-29T16:40:41Z

This feature is awesome!

Does it support generating a UAA server? Whenever I use an applicationType of uaa, I get:

Error: The application must be valid in order to be added to the JDL object.
Errors: No UAA application folder path
ERROR! Error while parsing applications and entities from the JDL Error: The application must be valid in order to be added to the JDL object.
Errors: No UAA application folder path

Here is my JDL:

application {
  config {
    applicationType uaa,
    baseName uaa,
    packageName com.test.uaa,
    authenticationType uaa,
    prodDatabaseType mssql,
    serviceDiscoveryType eureka,
    buildTool gradle
  }
}

deepu105 · 2018-09-29T20:49:36Z

@jsm174 you need to provide uaaBaseName property for uaa apps, please see https://www.jhipster.tech/jdl/#application_options

mraible · 2018-09-29T21:05:00Z

Maybe we should add something like Maven’s effective pom plug-in. https://maven.apache.org/plugins/maven-help-plugin/effective-pom-mojo.html You can use it to see your full pom.xml with your model hydrated with all the defaults. We could make it as simple as a —debug flag.

…

On Sep 29, 2018, at 14:49, Deepu K Sasidharan ***@***.***> wrote: @jsm174 you need to provide uaaBaseName property for uaa apps, please see https://www.jhipster.tech/jdl/#application_options — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

jsm174 · 2018-10-01T17:28:02Z

@jsm174 you need to provide uaaBaseName property for uaa apps, please see https://www.jhipster.tech/jdl/#application_options

Thank you! I totally missed uaaBaseName. Its working great!

jsm174 · 2018-10-02T17:36:32Z

Creating a UAA based setup with a gateway that has no database and has access to entities in separate microservices will cause the generator to display several warnings. The setup does work, and seems valid:

jhipster import-jdl app.jdl
Using JHipster version installed globally
Executing import-jdl app.jdl
Options: from-cli: true
The JDL is being parsed.
warn: Multiple applications are declared to have entity 'A', check field type to ensure compatibility.
warn: Multiple applications are declared to have entity 'B', check field type to ensure compatibility.
warn: Multiple applications are declared to have entity 'C', check field type to ensure compatibility.
warn: Multiple applications are declared to have entity 'D', check field type to ensure compatibility.
Found entities: A, B, C, D.
The JDL has been successfully parsed
Generating 4 applications.
Generating application uaa in a new parallel process
Generating application gateway in a new parallel process
Generating application microservicea in a new parallel process
Generating application microserviceb in a new parallel process
Executing jhipster:app on /Users/jsm174/git/optimo/intelligent-addressing/ia-backend2/newapp/uaa
Options: fromCli: true, withEntities: true, from-cli: true, with-entities: true
Executing jhipster:app on /Users/jsm174/git/optimo/intelligent-addressing/ia-backend2/newapp/microservicea
Options: fromCli: true, withEntities: true, from-cli: true, with-entities: true
Executing jhipster:app on /Users/jsm174/git/optimo/intelligent-addressing/ia-backend2/newapp/gateway
Options: fromCli: true, withEntities: true, from-cli: true, with-entities: true
Executing jhipster:app on /Users/jsm174/git/optimo/intelligent-addressing/ia-backend2/newapp/microserviceb
Options: fromCli: true, withEntities: true, from-cli: true, with-entities: true

app.jdl:

application {
  config {
    baseName uaa,
    applicationType uaa,
    packageName com.example.uaa,
    authenticationType uaa,
    uaaBaseName "uaa",
    prodDatabaseType postgresql,
    serviceDiscoveryType eureka,
    buildTool gradle
  }
}

application {
  config {
    baseName gateway,
    applicationType gateway,
    authenticationType uaa,
    packageName com.example.gateway,
    uaaBaseName "uaa",
    databaseType no,
    serviceDiscoveryType eureka,
    buildTool gradle
  }
  entities EntityA, EntityB, EntityC,EntityD
}

application {
  config {
    baseName microservicea,
    applicationType microservice,
    authenticationType uaa,
    packageName com.example.microservicea,
    uaaBaseName "uaa",
    prodDatabaseType postgresql,
    serviceDiscoveryType eureka,
    buildTool gradle
  }
  entities EntityA, EntityB
}

application {
  config {
    baseName microserviceb,
    applicationType microservice,
    packageName com.example.microserviceb,
    authenticationType uaa,
    uaaBaseName "uaa",
    prodDatabaseType postgresql,
    serviceDiscoveryType eureka,
    buildTool gradle
  }
  entities EntityC, EntityD
}

entity EntityA {
    data String
}

entity EntityB {
    data String
}

entity EntityC {
    data String
}

entity EntityD {
    data String
}

dto * with mapstruct
service * with serviceClass

microservice EntityA, EntityB with microservicea
microservice EntityC, EntityD with microserviceb

deepu105 requested a review from MathieuAA September 15, 2018 19:04

MathieuAA reviewed Sep 15, 2018

View reviewed changes

MathieuAA approved these changes Sep 15, 2018

View reviewed changes

mraible previously requested changes Sep 15, 2018

View reviewed changes

deepu105 added 10 commits September 16, 2018 12:40

core: minor typo correction

b3530f7

cli: add import-jdl

8f77a26

core: deprecate import-jdl yeoman subgen

34655d2

core: add js-object-pretty-print

42ee4f4

cli: use cliOnly when available

5d4b0c5

cli: add help for import

aefc540

cli: import-jdl for multiple apps

5ec334c

cli: import entities and apps correctly

9ca7bec

cli: add todo

fe4dd28

cli: move tests to folder

e5bdaf3

deepu105 added 5 commits September 16, 2018 12:40

core: review comment update

2c7e3b4

cli: improve import-jdl handling

a40415c

cli: move common methods to util

6e5e12c

test: fix cli test

ef833b5

test: cli: import-jdl tests

e9a1703

deepu105 force-pushed the 8185 branch from 51a3b5a to e9a1703 Compare September 16, 2018 10:42

cli: minor typo fix

f43a9bb

deepu105 mentioned this pull request Sep 16, 2018

Multiple application generation using JDL issues and improvements jhipster/jhipster-core#259

Closed

8 tasks

travis: upadte script flags

865512d

test: reorder ttest spec order

b3a5ef1

deepu105 merged commit eb3ed47 into jhipster:master Sep 16, 2018

jdubois added this to the 5.3.2 milestone Sep 17, 2018

jdubois changed the title ~~Full microservice stack gen using import-jdl~~ Full microservice stack generation using import-jdl Sep 17, 2018

deepu105 deleted the 8185 branch September 17, 2018 17:14

deepu105 mentioned this pull request Sep 21, 2018

Enriching the JDL grammar jhipster/jhipster-core#141

Closed

Uh oh!

Full microservice stack generation using import-jdl #8335

Full microservice stack generation using import-jdl #8335

Uh oh!

Conversation

deepu105 commented Sep 15, 2018 • edited by MathieuAA Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deepu105 commented Sep 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deepu105 commented Sep 15, 2018

Uh oh!

deepu105 commented Sep 15, 2018

Uh oh!

PierreBesson commented Sep 15, 2018

Uh oh!

MathieuAA commented Sep 15, 2018

Uh oh!

deepu105 commented Sep 15, 2018

Uh oh!

deepu105 commented Sep 15, 2018

Uh oh!

MathieuAA commented Sep 15, 2018

Uh oh!

deepu105 commented Sep 15, 2018

Uh oh!

MathieuAA Sep 15, 2018

Choose a reason for hiding this comment

Uh oh!

deepu105 Sep 15, 2018

Choose a reason for hiding this comment

Uh oh!

mraible commented Sep 15, 2018

Uh oh!

mraible left a comment

Choose a reason for hiding this comment

Uh oh!

mraible Sep 15, 2018

Choose a reason for hiding this comment

Uh oh!

mraible Sep 15, 2018

Choose a reason for hiding this comment

Uh oh!

mraible commented Sep 15, 2018

Uh oh!

mraible commented Sep 16, 2018

Uh oh!

mraible commented Sep 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deepu105 commented Sep 16, 2018 via email

Uh oh!

MathieuAA commented Sep 16, 2018 via email

Uh oh!

deepu105 commented Sep 16, 2018

Uh oh!

deepu105 commented Sep 16, 2018

Uh oh!

pascalgrimaud commented Sep 16, 2018

Uh oh!

deepu105 commented Sep 16, 2018

Uh oh!

deepu105 commented Sep 16, 2018

Uh oh!

MathieuAA commented Sep 16, 2018

Uh oh!

deepu105 commented Sep 19, 2018

Uh oh!

jdubois commented Sep 19, 2018

Uh oh!

deepu105 commented Sep 19, 2018 via email

Uh oh!

jsm174 commented Sep 29, 2018

Uh oh!

deepu105 commented Sep 29, 2018

Uh oh!

mraible commented Sep 29, 2018 via email

Uh oh!

jsm174 commented Oct 1, 2018

deepu105 commented Sep 15, 2018 •

edited by MathieuAA

Loading

deepu105 commented Sep 15, 2018 •

edited

Loading

mraible commented Sep 16, 2018 •

edited

Loading

jsm174 commented Oct 2, 2018 •

edited

Loading