Pages

Friday, September 21, 2018

SQL query or PL/SQL function to count business days between two dates

The weekend and holidays should be excluded. It is easy to exclude weekends, but the holiday are specific to each place and should be stored in an additional table. I store the holidays in a table HOLIDAYS_VD:

Select the time interval between two dates

The SQL query is adapted from here.

 select count(*) from (
--  select * from (
 select * from (
 select  dayinbetween, to_char( dayinbetween, 'DY' ) weekday from (
 select   startday+level-1 dayinbetween from (
 select startday ,endday-startday diffdays from (
 select   to_date('01.02.2000','DD.MM.YYYY') endday, to_date('29.02.2000','DD.MM.YYYY') startday from dual
 )
 ) connect by level <= diffdays+1
 )
 )where weekday not in ( 'SAT', 'SUN' )--all not weeknds in between
 ) a
 left join  holidays_vd h on a.dayinbetween=h.day
 where h.day is null
 order by dayinbetween; 

It is almost right, but it does not return 0 is the two dates are the same. I did not corrected it and abandoned it because I needed a PL/SQL function. It is below, it is based on the query above and it seems to work fine.

PL/SQL function subtracting dates and exluding weekends and holidays
create or replace PACKAGE indicators AS
TYPE DateListType IS   TABLE OF DATE;
function subtract_business_dates(    startdate DATE,    enddate   DATE) return integer;
END  ;
create or replace PACKAGE BODY  indicators AS
 function subtract_business_dates(
    startdate DATE,
    enddate   DATE) return integer
IS
  result INTEGER:=1;
 
  dateList DateListType:=DateListType();
  difference_days INTEGER;
BEGIN
  difference_days:=trunc(enddate)-trunc(startdate);
  if difference_days<0 then
  RAISE_APPLICATION_ERROR(-20000, 'The first day is after the second date');
  end if;
    if difference_days=0 then
 return 0;
  end if;
 
  dateList.extend(difference_days);
  FOR i IN dateList.first..dateList.last
  LOOP
    dateList(i):=startdate+i;
 
  END LOOP;
  
    select count(*) into result from (
 
   select * from (
    select  dayinbetween, to_char( dayinbetween, 'DY' ) weekday from (
   select  COLUMN_VALUE dayinbetween  from (table(dateList)) 
      )
    )where weekday not in ( 'SAT', 'SUN' )--all not weeknds in between
   ) a
   left join  holidays_vd h on a.dayinbetween=h.day
    where h.day is null
 
   ;
   return result;
END;
END  ;

Some tests to make sure that the function works well. Note, the table with holidays includes a test date 14.02.2000.

select indicators.subtract_business_dates(to_date('01.02.2000','DD.MM.YYYY'),to_date('29.02.2000','DD.MM.YYYY')) from dual;
select indicators.subtract_business_dates(to_date('04.02.2000','DD.MM.YYYY'),to_date('01.02.2000','DD.MM.YYYY')) from dual; --exception

select indicators.subtract_business_dates(to_date('03.02.2000','DD.MM.YYYY'),to_date('03.02.2000','DD.MM.YYYY')) from dual;--thu
select indicators.subtract_business_dates(to_date('03.02.2000','DD.MM.YYYY'),to_date('04.02.2000','DD.MM.YYYY')) from dual;--fri
select indicators.subtract_business_dates(to_date('03.02.2000','DD.MM.YYYY'),to_date('05.02.2000','DD.MM.YYYY')) from dual;--sat
select indicators.subtract_business_dates(to_date('03.02.2000','DD.MM.YYYY'),to_date('06.02.2000','DD.MM.YYYY')) from dual;--sun
select indicators.subtract_business_dates(to_date('03.02.2000','DD.MM.YYYY'),to_date('07.02.2000','DD.MM.YYYY')) from dual;--mon
select indicators.subtract_business_dates(to_date('03.02.2000','DD.MM.YYYY'),to_date('08.02.2000','DD.MM.YYYY')) from dual;--tue


select indicators.subtract_business_dates(to_date('04.02.2000','DD.MM.YYYY'),to_date('04.02.2000','DD.MM.YYYY')) from dual;--fri
select indicators.subtract_business_dates(to_date('04.02.2000','DD.MM.YYYY'),to_date('05.02.2000','DD.MM.YYYY')) from dual;--sat
select indicators.subtract_business_dates(to_date('04.02.2000','DD.MM.YYYY'),to_date('06.02.2000','DD.MM.YYYY')) from dual;--sun
select indicators.subtract_business_dates(to_date('04.02.2000','DD.MM.YYYY'),to_date('07.02.2000','DD.MM.YYYY')) from dual;--mon
select indicators.subtract_business_dates(to_date('04.02.2000','DD.MM.YYYY'),to_date('08.02.2000','DD.MM.YYYY')) from dual;--tue

select indicators.subtract_business_dates(to_date('10.02.2000','DD.MM.YYYY'),to_date('10.02.2000','DD.MM.YYYY')) from dual;--thu
select indicators.subtract_business_dates(to_date('10.02.2000','DD.MM.YYYY'),to_date('11.02.2000','DD.MM.YYYY')) from dual;--fri
select indicators.subtract_business_dates(to_date('10.02.2000','DD.MM.YYYY'),to_date('12.02.2000','DD.MM.YYYY')) from dual;--sat
select indicators.subtract_business_dates(to_date('10.02.2000','DD.MM.YYYY'),to_date('13.02.2000','DD.MM.YYYY')) from dual;--sun
select indicators.subtract_business_dates(to_date('10.02.2000','DD.MM.YYYY'),to_date('14.02.2000','DD.MM.YYYY')) from dual;--mon holiday
select indicators.subtract_business_dates(to_date('10.02.2000','DD.MM.YYYY'),to_date('15.02.2000','DD.MM.YYYY')) from dual;--tue
select indicators.subtract_business_dates(to_date('10.02.2000','DD.MM.YYYY'),to_date('16.02.2000','DD.MM.YYYY')) from dual;--wed

Thursday, May 24, 2018

Copying data from Oracle to MySQL or from any to any database

There is no official tool to export data from Oracle to MySQL database. Recurrently I need to copy a huge table from the source Oracle to MySQL database. It seems to me that configuring any available tools is more complicated than just using a very simple and therefore reliable java class. The class simply executes in the target database an INSERT statement for each row of the SELECT query executed in the source database. I execute such an tiny application as a background job to copy millions of rows from one VM to another overnight.

The prerequisite is that the target table exists in the target database. But DDL is the easiest part of an export.

public class Main {

    static String SOURCE_TABLE_NAME = "ONM_MOL_DATA", TARGET_TABLE_NAME = "ONM_MOL_DATA_COPY";
    String SQL = "select * from " + SOURCE_TABLE_NAME; 
    String INSERT = "insert into " + TARGET_TABLE_NAME + " values()";
    String DELETE = "delete from " + TARGET_TABLE_NAME;

    void run() throws SQLException, IOException {

        try (Connection oracleCon = ConnectionFactory.getOracleConnection();
                Connection mySQLCon = ConnectionFactory.getMySQLConnection();
                Statement oracleStmt = oracleCon.createStatement();
                ResultSet oracleRs = oracleStmt.executeQuery(SQL)) {
            ResultSetMetaData md = oracleRs.getMetaData();

            String mySQLInsert = getInsert(md.getColumnCount());

            // clean
            Statement mySQLDeleteStmt = mySQLCon.createStatement();
            mySQLDeleteStmt.executeUpdate(DELETE);

            // copy
            mySQLCon.setAutoCommit(false);
            System.out.println(mySQLInsert);
            PreparedStatement mySQLInsertStmt = mySQLCon.prepareStatement(mySQLInsert);

            int row = 0;
            while (oracleRs.next()) {
                for (int c = 1; c <= md.getColumnCount(); c++) {
                    mySQLInsertStmt.setObject(c, oracleRs.getObject(c));
                }
                mySQLInsertStmt.executeUpdate();
                row++;
                if (row % 100000 == 0) {
                    System.out.println("row=" + row);
                }
            }
            mySQLCon.commit();
        }

    }

    String getQuestionMarks(int count) {
        String[] ar = new String[count];
        Arrays.fill(ar, "?");
        return String.join(",", ar);
    }

    String getInsert(int count) {
        return INSERT.replace("values()", "values(" + getQuestionMarks(count) + ")");

    }

    public static void main(String... args) throws SQLException, IOException {
        new Main().run();
    }
}

There is nothing special in ConnectionFactory class:

public class ConnectionFactory {

    public static Connection getMySQLConnection() throws SQLException {
         return DriverManager.getConnection(MYSQL_URL);
    }

    public static Connection getOracleConnection() throws SQLException {
        return DriverManager.getConnection(ORACLE_URL);
    }
}

Executing a jar as a background job on Linux

Often I need to leave overnight a long-running Java application. This is done with help of nohup command that blocks SIGHUP signals to the preceeded application and thereby prevents the application from exiting when the terminal session is terminated:

nohup java -jar CopyOracleToMySQL-1.0.jar > copy.out 2>&1 &

The STDOUT and STDERR output is saved to the file copy.out.

Tuesday, May 8, 2018

Copying a table to another MySQL database

Unfortunately, in MySQL there are no decent export tools similar to Data Pump in Oracle. Luckily one can copy either the entire database data folder or only the required individual tables. The text is adapted from the MySQL manual.

  1. In the source database, first generate the CREATE statement for the target table so that an identical table can be created in the target database. Then execute an SQL command:

    FLUSH TABLES TEMP FOR EXPORT;

    A TEMP.cfg file is created in the MySQL data directory. This file has to be copied to the target machine. I first copy it to /tmp, so that it becomes more accessible.

    sudo cp TEMP.{ibd,cfg} /tmp
    cd /tmp
    chmod 644 TEMP.*
    

    Then execute an SQL command. The previously created TEMP.cfg disappears.

    UNLOCK TABLES;
  2. In the target database, execute SQL to create an identical table using the DDL from the source database and discard its tablespace:

    CREATE TABLE `TEMP` (
      `INTERVENTION_LIBELLE` varchar(93) NOT NULL,
      ...
      KEY `INTERVENTION_LIBELLE_idx` (`INTERVENTION_LIBELLE`),
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
    ALTER TABLE TEMP DISCARD TABLESPACE;

    Copy the files from the source machine to the local MySQL data folder:

    cd /data1/mysql/axya/
    scp test@source-machine:/tmp/TEMP.{ibd,cfg} .
    chmod 640 TEMP.*
    chown mysql:mysql TEMP.*
    

    Execute the last SQL command after which the data becomes usable.

    ALTER TABLE TEMP IMPORT TABLESPACE;

How to cast varchar to int or datetime in MySQL

To convert a varchar value into int:
update TEMP set PATIENT_ID=cast(PATIENT_ID_ORIG as UNSIGNED);
To convert a varchar value into datetime

One need to use a troublesome function STR_TO_DATE. The problem with this function is that it never produces errors, instead it produces NULL and a warning that has to be revealed by an additional statement:

SELECT STR_TO_DATE('05/11/2012 08:30:00','%d/%m/%Y %H:%i:%s');
2012-11-05 08:30:00

SELECT STR_TO_DATE('505/11/2012 08:30:00','%d/%m/%Y %H:%i:%s');
null

SHOW WARNINGS;
Warning 1411 Incorrect datetime value: '505/11/2012 08:30:00' for function str_to_date

There might also be crazy conversion to zero dates. I converted valid values like that:

SET  sql_mode = 'ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,STRICT_ALL_TABLES';
SELECT @@SESSION.sql_mode;
ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

update TEMP set ENTREE_SALLE =STR_TO_DATE(ENTREE_SALLE_ORIG,'%d/%m/%Y %H:%i:%s');

Tuesday, April 24, 2018

How to recover passwords in SQL developer

I just want to note down and share a useful link https://github.com/maaaaz/sqldeveloperpassworddecryptor with a python script that not only contains a comprehensive user guide but also works. Before stumbling upon it, I also tried some SQL developer plugins that turned out not to work.

Monday, April 23, 2018

Change schema in Oracle

Even though I do it regularly, I always forget the commands.

To display the current user and the current schema:

SELECT sys_context('USERENV','SESSION_USER') as "USER NAME", sys_context('USERENV', 'CURRENT_SCHEMA') as "CURRENT SCHEMA" FROM dual;

To change the current schema, instead of prefixing the table names:

ALTER SESSION SET CURRENT_SCHEMA = hdm;

Add MySQL or Oracle driver to create a datasource in Wildfly 10

In any other server one simply puts any jar into the library folder to make included in the classpath. In contrast, in Wildfly there is no such a folder - it is based on a modular classloading architecture. One needs to create an individual folder for any additional jar, which is called a module. The official documentation recommends including the required jar in the web archive instead of creating modules. However, this is impossible if one needs to create as datasource, which depends on a driver in the server classpath (A datasource can alternatively be deployed, which I find inconvenient).

  1. Create a new folder path$JBOSS_HOME/modules/system/layers/base/com/mysql/main

    Inside the created folder, create a file module.xml with contents:

    <?xml version="1.0" encoding="UTF-8"?>
    <module xmlns="urn:jboss:module:1.3" name="com.mysql">
         <resources>
            <resource-root path="mysql-connector-java-5.1.42-bin.jar"/>
         </resources>
          <dependencies>
       <module name="marian.mysqllogger"/> 
            <module name="javax.api"/> 
            <module name="javax.transaction.api"/>
        </dependencies>
    </module>

    Note, module marian.mysqllogger is obviously optional, it logs the executed SQL commands. It is described in my previous posts.

  2. Create a new folder path$JBOSS_HOME/modules/system/layers/base/com/oracle/main

    Inside the created folder, create a file module.xml with contents:

    <?xml version="1.0" encoding="UTF-8"?>
    <module xmlns="urn:jboss:module:1.3" name="com.oracle">
         <resources>
            <resource-root path="ojdbc6.jar"/>
         </resources>
    
        <dependencies>
            <module name="javax.api"/> 
            <module name="javax.transaction.api"/>
        </dependencies>
    </module>
  3. Copy a MySQL driver jar indicated in the xml file (e.g. mysql-connector-java-5.1.42-bin.jar) into folder $JBOSS_HOME/modules/system/layers/base/com/mysql/main. The oracle driver indicated in the xml file (e.g. ojdbc6.jar) should also be placed into the folder $JBOSS_HOME/modules/system/layers/base/com/oracle/main with the xml file.

  4. Finally, the datasources can be created in the Administration Console after the Wildfly is restarted.

    Alternatively, you can directly edit $JBOSS_HOME/standalone/configuration/standalone.xml. Extend the part dedicated to datasources:

            <subsystem xmlns="urn:jboss:domain:datasources:4.0">
                <datasources>
                     <datasource jta="true" jndi-name="java:/OracleDS" pool-name="OracleDS" enabled="true" use-ccm="true">
                        <connection-url>jdbc:oracle:thin:@localhost:1521:orcl2</connection-url>
                        <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
                        <driver>oracle</driver>
                        <security>
                            <user-name>username</user-name>
                            <password>password</password>
                        </security>
                        <validation>
                            <valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker"/>
                            <background-validation>true</background-validation>
                            <stale-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleStaleConnectionChecker"/>
                            <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleExceptionSorter"/>
                        </validation>
                    </datasource>
                    <datasource jta="true" jndi-name="java:/MySqlDS" pool-name="MySqlDS" enabled="true" use-ccm="true">
                        <connection-url>jdbc:mysql://localhost:3306/wildfly?useSSL=false&profileSQL=true&logger=com.mysql.jdbc.log.MySlf4JLogger</connection-url>
                        <driver-class>com.mysql.jdbc.Driver</driver-class>
                        <driver>mysql</driver>
                        <security>
                            <user-name>username</user-name>
                            <password>password</password>
                        </security>
                        <validation>
                            <valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLValidConnectionChecker"/>
                            <background-validation>true</background-validation>
                            <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter"/>
                        </validation>
                    </datasource>
                    <drivers>
                        <driver name="oracle" module="com.oracle">
                            <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
                        </driver>
                        <driver name="mysql" module="com.mysql">
                            <driver-class>com.mysql.jdbc.Driver</driver-class>
                        </driver>
                    </drivers>
                </datasources>
            </subsystem>

    com.mysql.jdbc.log.MySlf4JLogger is a handy logger of executed SQL statements. It is loaded from marian.mysqllogger module.

    Restart the Wildfly.

Monday, March 26, 2018

Loading a long value into a String variable from a text file

The max length of a string seems to equal Integer.MAX_VALUE. However, a string literal length in Java is represented by two bytes implying that it cannot be over 65535 bytes. If you try to compile a class with a longer string, an error constant string too long will occur. Sometimes, for example for tests, one needs to use longer String values. Such values can be loaded from a file.

Suppose, a String variable has to be assigned the entire contents of a output.xml file, which contains ~400,000 characters and is saved in the classpath. Method inputStreamToString loads the contents from an InputStream into a String variable:

public class FileUtils {

    public String inputStreamToString(InputStream is) throws IOException {
        StringBuilder sb = new StringBuilder();
        char[] buffer = new char[1024 * 8];
        try (BufferedReader in = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8))) {
            int length;
            while ((length = in.read(buffer)) > -1) {
                sb.append(buffer, 0, length);
            }
        }
        return sb.toString();
    }
}

A test class:

public class FileUtilsTest {

    FileUtils i = new FileUtils();

    @Test
    public void testInputStreamToStringFromResourceFile() throws IOException {
        String resp = i.inputStreamToString(getClass().getResourceAsStream("/output.xml"));
        System.out.println(resp.length());// 358,830
        assertEquals(resp.length(), 358779);
    }

    @Test
    public void testInputStreamToStringFromString() throws IOException {
        System.out.println(Charset.defaultCharset()); // windows-1252
        String str = "this is a test string to be converted to InputStream";
        String copy = i.inputStreamToString(new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8)));
        assertEquals(str.length(), copy.length());
    }
}

Another sorter possibility:

    public String readFileToString(String fileName) throws IOException {
        Path filePath = getPathInResources(fileName);
        return new String(Files.readAllBytes(filePath), StandardCharsets.UTF_8);
    }
How to convert a String to an InputStream
InputStream is=new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8));

The code is used in the second test above.

How to save a String to a file
 Files.write(Paths.get("src\\test\\resources\\xml0.xml"),str.getBytes(StandardCharsets.UTF_8));

Wednesday, March 14, 2018

Changing font size and default language in SQL developer or Data Integrator

Changing the default tiny font size

By default the letters in SQL developer or Data integrator are hardly visible. The steps to increase the font size are quite the same on windows and linux. In %userprofile%/AppData on Windows or $HOME on Linux search for a file ide.properties. One file will be found in SQL developer folder, whereas in ODI folder two files will be found (No idea why because it does not matter).

In SQL developer's file uncomment a line with Ide.FontSize=18 and set the convienient font size (no less than 18).

Add the the same line to the two ODI files which seems identical on Linux, and different on Windows (the last two pictures).

Restart the applications.

Changing the default language to English

Java uses the default locale of the computer. To change the language, one needs to change JVM system variables. It can be done in a configuration file ide.conf located in the installation folder of SQL developer or ODI, e.g C:\oracle\Middleware\Oracle_Home. Add to lines the end of the file and then restart the application:

AddVMOption -Duser.language=en
AddVMOption -Duser.country=US

Thursday, March 8, 2018

Adding a datasource to Tomcat and specifying it in persistence.xml

To add a datasource connecting to Oracle database, save the Oracle driver ojdbc7.jar into CATALINA_HOME/lib. Then add a line with the connection details into CATALINA_HOME/conf/context.xml:

<Resource name="jdbc/saphirOracleDB" auth="Container" type="javax.sql.DataSource"
        maxTotal="20" maxIdle="30" maxWait="10000"
        username="username" password="password" driverClassName="oracle.jdbc.OracleDriver"
        url="jdbc:oracle:thin:@//hostname:1555/servicename"/>

In a sample persistence.xml depending on the created datasource, the reference to the datasource is obtained using JNDI name java:/comp/env/jdbc/saphirOracleDB

<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.1" xmlns="http://xmlns.jcp.org/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/persistence http://xmlns.jcp.org/xml/ns/persistence/persistence_2_1.xsd">
  <persistence-unit name="Saphir" transaction-type="RESOURCE_LOCAL">
    <provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>
    <non-jta-data-source>java:/comp/env/jdbc/saphirOracleDB</non-jta-data-source>
    <class>entities.sqlresultmapping.DummyForMapping</class>
  </persistence-unit>
</persistence>

The entity manager can be obtained in the web application by using the specified persitence unit name:

    private static EntityManagerFactory emf = Persistence.createEntityManagerFactory("Saphir");
    private static EntityManager em = emf.createEntityManager();

Removing accents and other diacritical marks from unicode text so as to convert it into English letters

Often I need to convert unicode text, e.g. French, into English letters. The general way to remove diacritical marks is to decompose characters into chars representing letters and separately marks using Normalizer with form NFD, and then remove all chars holding diacritical signs using a regular expression \p{InCombiningDiacriticalMarks}+ matching the "Combining Diacritical Marks" unicode character block.

The sample class below uses as the input a meaningless text made up of french words with various accents:

public class Clean   {

    static void describe(String str) {
        System.out.println(str + " " + str.length());
    }

    public static void main(String[] args) {
        String str = "«J'ai levé la tête. Il doit être français». Il n'a pensé à lui ôter l'âge et se met à nager âgé.";
        describe(str);
        String normalizedString = Normalizer.normalize(str, Normalizer.Form.NFD);
        // the regexp corresponds to Character.UnicodeBlock.COMBINING_DIACRITICAL_MARKS
        String noDiacriticalMarks = normalizedString.replaceAll("\\p{InCombiningDiacriticalMarks}+", "");
        describe(normalizedString);
        describe(noDiacriticalMarks);
    }
}

In the output the first line is the original string. The second is the same string but normalized. Note, the accents are stored as individual characters which are eliminated in the third line. Each line contains the length of the string.

«J'ai levé la tête. Il doit être français». Il n'a pensé à lui ôter l'âge et se met à nager âgé. 96
«J'ai levé la tête. Il doit être français». Il n'a pensé à lui ôter l'âge et se met à nager âgé. 107
«J'ai leve la tete. Il doit etre francais». Il n'a pense a lui oter l'age et se met a nager age. 96
Remove diacritical marks from a string in Javascript

Analogous approach in javascript taken from here:

const str = "Crème Brulée"
console.log(str + "=>" + str.normalize('NFD').replace(/[\u0300-\u036f]/g, ""));

Sunday, February 18, 2018

Using Backbone Model to post its attributes together with files as multipart form data

Backbone is a tiny and very flexible library for REST-based front end such as single page applications. But I did not find many examples of how to save a model containing both text values and selected files. As the data contains files, it has to be encoded as multipart form data including binary files and the text attributes in json format. A possible short Javascript code in the Backbone model is quite simple:

Backbone.Model.extend({
selectedFiles: [], // the photos to be uploaded
  
saveMultipart: function () {
    var formData = new FormData();
    selectedFiles.forEach(function (photo) {
        formData.append('photo', photo, photo.originalNameSize.nameWithoutExtension);
    });
    formData.append('dataObject', JSON.stringify(this.toJSON( )));
    var options = {
        data: formData,
        contentType: false
    };
    this.save(null, options);
}
// other useful functions
            
});

To submit the model together with the files stored in selectedFiles, method saveMultipart should be called instead of the usual save.

In a JAX-RS-based REST backend the json is extracted from the part arbitrarily named here dataObject and parsed into java classes, whereas the files from the remaining parts are processed in some other way.

A working sample application that is not at all Backbone-based but includes also an example of such a Backbone model-based data submission is stored here. It will be described in more detail in a post about client-side image resizing and uploading them to REST resources.

Saturday, February 17, 2018

Conversion between Date and LocalDateTime

I do not know why the conversion between Date and LocalDate is so complicated and ugly. But I have to do it increasingly often. So this note is on how to convert between Date, LocalDate and milliseconds.

public static void main(String[] args) {
    // convert from Date to LocalDateTime
    Date d = new Date();
    LocalDateTime ld = LocalDateTime.ofInstant(d.toInstant(), ZoneId.systemDefault());

    // convert from LocalDateTime to Date  
    Date d2 = Date.from(ld.atZone(ZoneId.systemDefault()).toInstant());
    System.out.println("date : " + d);
    System.out.println("ldate: " + ld);
    System.out.println("date2: " + d2);

    // compare milliseconds
    System.out.println("millis : " + d.getTime());
    System.out.println("lmillis: " + ld.atZone(ZoneId.systemDefault()).toInstant().toEpochMilli());
}

The output is:

date : Sat Feb 17 14:41:00 CET 2018
ldate: 2018-02-17T14:41:00.941
date2: Sat Feb 17 14:41:00 CET 2018

millis : 1518874860941
lmillis: 1518874860941

Thursday, February 1, 2018

CORS filter for JAX-RS

It should be registered as a singleton in the Application subclass.

public class CORSResponseFilter implements ContainerResponseFilter {

    Logger logger = LoggerFactory.getLogger(getClass().getName());

    @Override
    public void filter(ContainerRequestContext requestContext, ContainerResponseContext responseContext) throws IOException {
        MultivaluedMap headers = responseContext.getHeaders();

        String origin = requestContext.getHeaderString("Origin");
        if (origin != null) {
            headers.add("Access-Control-Allow-Origin", requestContext.getHeaderString("Origin"));
        } 

        headers.add("Access-Control-Allow-Methods", "GET, POST, DELETE, PUT");
        headers.add("Access-Control-Allow-Headers", "X-Requested-With, Content-Type, X-Codingpedia");
        headers.add("Access-Control-Allow-Credentials", true);
    }
}

RESTEasy also provides a CORS filter class. I do not know why it is not only response but also request filter. It is used as any other filter but needs configuration of all the headers to be added.

CorsFilter filter = new CorsFilter();
filter.getAllowedOrigins().add("*");

Configuring Jackson object mapper in RESTEasy

While transforming between Java classes and JSON, Jackson library considers both its own annotations and the conventional JAXB annotations. The final result maybe not obvious. Let's consider a sample class from a sample application:

@XmlRootElement
@XmlAccessorType(XmlAccessType.PUBLIC_MEMBER)
public class MyBean {

    String firstName, lastName, fullName;

    public MyBean(String firstName, String lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
    }

    public MyBean() {
    }

    @XmlElement(name = "jaxbFirstName")
    public String getFirstName() {
        return firstName;
    }

    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }

    @JsonProperty("jacksonLastName")
    public String getLastName() {
        return lastName;
    }

    public void setLastName(String lastName) {
        this.lastName = lastName;
    }

    @XmlElement(name = "jaxbFullName")
    @JsonProperty("jacksonFullName")
    public String getFullName() {
        return firstName + " " + lastName;
    }

    public void setFullName(String fullName) {
        this.fullName = fullName;
    }
}

Jackson set up by RESTEasy prefers its own annotations over the JAXB ones. Note, the default object mapper ignores JAXB annotations (see below). The default output of an object mapper will be:

{"jaxbFirstName":"John","jacksonLastName":"Smith","jacksonFullName":"John Smith"}
Configuring Jackson used by JAX-RS

To configure Jackson, one has to provide his own configured instance by means of a context provider implementing ContextResolver interface. The provider produces an ObjectMapper instance (according to the authors it can be reused) that is to be used by JAX-RS. The following class from another sample application provides a object mapper that produces nicely formatted JSON.

public class MyObjectMapperProvider implements ContextResolver {

    ObjectMapper objectMapper = createObjectMapper();

    @Override
    public ObjectMapper getContext(final Class type) {
        return objectMapper;
    }

    ObjectMapper createObjectMapper() {
        ObjectMapper objectMapper = new ObjectMapper();
        objectMapper.enable(SerializationFeature.INDENT_OUTPUT);
        return objectMapper;
    }
}

And the custom provider has to be registered as a singleton:

@ApplicationPath("/api")
public class MyApplication extends Application {

    public MyApplication() {
        singletons = new HashSet<Object>() {
            {
                add(new MyObjectMapperProvider());
            }
        };
        resources = new HashSet<Class<?>>() {
            {
                add(MyResource.class);
            }
        };
    }

    Set<Object> singletons;
    Set<Class<?>> resources;

    @Override
    // note, it is called twice during RESTEasy initialization, 
    public Set<Class<?>> getClasses() {
        System.out.println(">getClasses()");
        return resources;
    }

    @Override
    // note, it is called twice during RESTEasy initialization, 
    public Set<Object> getSingletons() {
        System.out.println(">getSingletons()");
        return singletons;
    }
}

The json received from the service is formatted now:

{
  "firstName" : "John",
  "jacksonLastName" : "Smith",
  "jacksonFullName" : "John Smith"
}

Note, unlike the default Jackson object mapper in RESTEasy, the default Jackson object mapper (created as above ObjectMapper objectMapper = new ObjectMapper() ) does not recognize JAXB annotations.

Enabling JAXB annotations in Jackson object mapper

The customized object mapper instance has to be further configured in the context provider shown above:

    ObjectMapper createObjectMapper() {
        ObjectMapper objectMapper = new ObjectMapper();
        objectMapper.enable(SerializationFeature.INDENT_OUTPUT).registerModule(new JaxbAnnotationModule());
        return objectMapper;
    }

Now JAXB annotations are priveleged over Jackson ones in the produced JSON:

{
  "jaxbFirstName" : "John",
  "jacksonLastName" : "Smith",
  "jaxbFullName" : "John Smith"
}
Disabling unconventional Jackson annotations

The customized object mapper instance has to be further configured in the context provider shown above:

    ObjectMapper createObjectMapper() {
        ObjectMapper objectMapper = new ObjectMapper();
        objectMapper.enable(SerializationFeature.INDENT_OUTPUT).setAnnotationIntrospector(new JaxbAnnotationIntrospector());;
        return objectMapper;
    }

Now Jackson are ignored in the produced JSON:

{
  "lastName" : "Smith",
  "jaxbFirstName" : "John",
  "jaxbFullName" : "John Smith"
}
Ignore empty properties during serialization

Another usefull setting feature preventing nulls and empty collections from being included into resulting json.

public class MyObjectMapperProvider implements ContextResolver {
    
    static ObjectMapper objectMapper = createObjectMapper();

    @Override
    public ObjectMapper getContext(final Class type) {
        return objectMapper;
    }
    
    static ObjectMapper createObjectMapper() {
        ObjectMapper objectMapper = new ObjectMapper();
        objectMapper.enable(SerializationFeature.INDENT_OUTPUT).setAnnotationIntrospector(new JaxbAnnotationIntrospector()).setSerializationInclusion(JsonInclude.Include.NON_EMPTY);
        return objectMapper;
    }
    
    public static ObjectMapper getObjectMapper() {
        return objectMapper;
    }
}

Sign in with Google into a web application using the server flow.

This post is based on the Google documentation on Google's OAuth 2.0 authentication, where OpenID Connect seems to be the most pertinent and comprehensive section. But overall the documentation is quite confusing. So I summarize it here. A sample Java web application is in GitHub.

First, obtain OAuth 2.0 credentials and set redirect URIs in the Google API Console:

Authentication comes down to obtaining an id token via HTTPS from Google. The most commonly used approaches for authenticating a user Google documentation calls the server/basic flow and the implicit flow:

  • The server/basic flow allows the back-end server of an application to identify the user.
  • The implicit flow is when a client-side JavaScript app accesses APIs directly and not via its back-end server.

The major difference is that in implicit flow tokens are sent as url hash, whereas in server flow tokens are sent as url parameters. Also unlike the implicit flow, the server flow requires client secret. Here I illustrate the server flow for authentication. The implicit flow using Google API Javascript library I demonstrated in a previous post.

When a user tries to sign in with Google, the application has to:

  1. Send an authentication request with the appropriate parameters to Google authorization_endpoint.
    • client_id from the API Console.
    • response_type should be code, which launches a Basic flow. If the value is token id_token or id_token token, launches an Implicit flow, requiring the use of Javascript at the redirect URI to retrieve tokens from the URI #fragment.
    • nonce A random value generated by your app that enables replay protection.
    • scope should be openid email. The scope value must begin with the string openid and then include profile or email or both.
    • redirect_uri the url to which browser will be redirected by Google after the user completes the authorization flow. The url must exactly match one of the redirect_uri values listed in the API Console. Even trailing slash / matters.
    • state should include the value of the anti-forgery unique session token, as well as any other information needed to recover the context when the user returns to your application, e.g., the starting URL.

    A sample URL from the link, which is supposed to be a button, my sample application:

    https://accounts.google.com/o/oauth2/v2/auth?client_id=517342657945-qv1ltq618ijj9edusgnnbpmbghkatc2q.apps.googleusercontent.com&redirect_uri=http://localhost:8080/test/server&scope=email&response_type=code&nonce=1845254249&state=u72lptpmf78lqv7nuid23l8hfa

    Google handles the user authentication and user consent. After a user signs in to Google, the browser is redirected to the indicated url with two appended parameters:

    http://localhost:8080/test/server?state=u72lptpmf78lqv7nuid23l8hfa&code=4/H3hLypL85UqpnKUT3po5vWIeYyZD4oPBjNyGk_rcYNI#

    Note, if a user has one Gmail account and is logged in, the user will not see any Google consent page and will be automatically redirected. But if the user has several accounts or is logged out, he has to choose one or log in on the Google page.

    If the user approves the access request, an authorization code is added to redirect_uri. Otherwise, the response contains an error message. Either authorization code or error message appear on the query string.

  2. Confirm that the state received from Google matches the state value sent in the original request.
  3. Exchange the authorization code for an access token and ID token.

    The response includes a one-time code parameter that can be exchanged for an access token and ID token. For that, the server sends POST request to the token_endpoint. The request must include the following parameters in the POST body:

    • code the received authorization code
    • client_id from the API Console
    • client_secret from the API Console
    • redirect_uri specified in the API Console
    • grant_type equals authorization_code

    A sample request by my sample application:

    POST https://www.googleapis.com/oauth2/v4/token
    
    code=4/H3hLypL85UqpnKUT3po5vWIeYyZD4oPBjNyGk_rcYNI&client_id=517342657945-qv1ltq618ijj9edusgnnbpmbghkatc2q.apps.googleusercontent.com&client_secret=0PVwYgPuLpH3PFHljPkbtJeP&redirect_uri=http://localhost:8080/test/server&grant_type=authorization_code

    A successful response includes a JSON with fields:

    • access_token A token that can be sent to a Google API.
    • id_token containing the information about the user
    • expires_in The remaining lifetime of the access token.
    • token_type always has the value Bearer.

    The response to the request above was:

    {"access_token":"ya29.GlxTBVKmo1YcUb_gyYstxB1Q-YpYzVviVp-uJKvU6CNfyhGUtD8oZJhliX9YADuKjebSZFxK1yL--TRxW_POT5vyBh9L43tlmzERrU8cwSSkl9U3n0zkY4nbHcnvoA","token_type":"Bearer","expires_in":3600,"id_token":"eyJhbGciOiJSUzI1NiIsImtpZCI6IjI2YzAxOGIyMzNmZTJlZWY0N2ZlZGJiZGQ5Mzk4MTcwZmM5YjI5ZDgifQ.eyJhenAiOiI1MTczNDI2NTc5NDUtcXYxbHRxNjE4aWpqOWVkdXNnbm5icG1iZ2hrYXRjMnEuYXBwcy5nb29nbGV1c2VyY29udGVudC5jb20iLCJhdWQiOiI1MTczNDI2NTc5NDUtcXYxbHRxNjE4aWpqOWVkdXNnbm5icG1iZ2hrYXRjMnEuYXBwcy5nb29nbGV1c2VyY29udGVudC5jb20iLCJzdWIiOiIxMTY4NjczMDQ5NzEyODQ3OTg1NjYiLCJlbWFpbCI6Im1hcmlhbi5jYWlrb3Zza2lAZ21haWwuY29tIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsImF0X2hhc2giOiJLQ2tlX051V1NXMzRLYng5XzhtRm9BIiwibm9uY2UiOiIxODQ1MjU0MjQ5IiwiZXhwIjoxNTE3NDM2MjYwLCJpc3MiOiJodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20iLCJpYXQiOjE1MTc0MzI2NjB9.MR6tnc5qnnL1VZmmONp4brWj5C9FQqIopIqa-UPc9NMz_qbAP37VpCSLn3CDdUouXCG7XpjTQbXZRvg9ZUP9-v_J5K9Crgp75csrVIBoWVre_yjncoFusEAc0efQOLUFwyKLV6cUsTMiUVoAkwWG6tOe_ZwXshq3-psblqpwwJxyILFFE2QiJviLH622S9YPBv0LA-tdTeqXOzt7yAK_cBeY-dnXXJfwVErY0yCGFAGOWf3WtTtzaWzxBshpMae9jRazsd3qgyiVtahD8IlfPUHhpYzDZs0RKuXBbfBF_wuB-cfyNhtuAdJfdaVNcWGLqqkQ4qkJGFhP8L5VMe1U_g"}
    
  4. Obtain user information from the ID token

    An ID Token is a JWT (JSON Web Token) - a signed Base64-encoded JSON object. Since it is received directly from Google over HTTPS it does not need to be validated. The encoded JSON contains the following fields:

    • email The user's email address provided only if your scope included email
    • profile The URL of the user's profile page provided when scope included profile
    • name The user's full name, in a displayable form provided when scope included profile
    • nonce The value of the nonce supplied by your app in the authentication request. You should enforce protection against replay attacks by ensuring it is presented only once.

    A simple Java code to extract the email from an id token:

    public JsonObject decodeIdToken(String idToken) {
        String secondStr = idToken.split("\\.")[1];
        byte[] payloadBytes = Base64.getDecoder().decode(secondStr);
        String json = new String(payloadBytes);
        JsonReader jsonReader = Json.createReader(new StringReader(json));
        return jsonReader.readObject();
    }

    The id token above was decoded by this method into:

    {"azp":"517342657945-qv1ltq618ijj9edusgnnbpmbghkatc2q.apps.googleusercontent.com","aud":"517342657945-qv1ltq618ijj9edusgnnbpmbghkatc2q.apps.googleusercontent.com","sub":"116867304971284798566","email":"marian.caikovski@gmail.com","email_verified":true,"at_hash":"KCke_NuWSW34Kbx9_8mFoA","nonce":"1845254249","exp":1517436260,"iss":"https://accounts.google.com","iat":1517432660}
    
  5. Authenticate the user in your application.

To keep my sample application simple, all those steps are done in a servlet. If any check fails, a error message is displayed with a link for sign in. If user successfully logs in, his email is displayed. The email suffices for authentication in the back end.

@WebServlet(name = "MyAuthServlet", urlPatterns = {"/server"})
public class MyAuthServlet extends HttpServlet {

    OpenId openId = new OpenId();

    protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        response.setContentType("text/html;charset=UTF-8");
        String code = request.getParameter(CODE);
        HttpSession session = request.getSession();
        String receivedState = request.getParameter(STATE); 
        String savedState = (String) session.getAttribute(STATE);
        String newState = openId.getState();
        session.setAttribute(STATE, newState);
        String savedNonce = (String) session.getAttribute(NONCE);
        String newNonce = openId.getNonce();
        session.setAttribute(NONCE, newNonce);

        try (PrintWriter out = response.getWriter()) {
            if (code != null) {
                if (savedState.equals(receivedState)) {
                    String idToken = openId.exchangeCodeForToken(code);
                    if (idToken != null) {
                        JsonObject json = openId.decodeIdToken(idToken);
                        String receivedNonce = json.getString(NONCE);
                        if (savedNonce.equals(receivedNonce)) {
                            String email = json.getString(EMAIL);
                            out.println("<p>Hello " + email + "</p>");
                            return;
                        } else {
                            out.println("Nonces differ");
                        }
                    } else {
                        out.println("Id token is missing");
                    }
                } else {
                    out.println("States are different");
                }
            } else {
                out.println("<p>Code is null</p>");
            }
            out.println("<a href='" + openId.getUrl(newState, newNonce) + "'>Click to sign in</a>");
        }
    }
}

Wednesday, January 31, 2018

POST with HttpUrlConnection

A shot note on POST requests:

public InputStream post(String url, String params) throws IOException {
    URL u = new URL(url);
    HttpURLConnection con = (HttpURLConnection) u.openConnection();
    con.setRequestMethod("POST");
    con.setDoOutput(true);
    try (OutputStreamWriter out = new OutputStreamWriter(con.getOutputStream())) {
        out.write(params);
    }

    return con.getInputStream();
}

GET request much simpler:

    String getAmazonHostName() throws  IOException {
        URL url = new URL("http://169.254.169.254/latest/meta-data/public-hostname");
        try (BufferedReader in = new BufferedReader( new InputStreamReader(url.openStream()))) {
            String inputLine = in.readLine();
            System.out.println("amazon public hostname: " + inputLine);
            return inputLine;
        }
    }

Tuesday, January 30, 2018

Switching off automatic discovery of resource classes and providers in JAX-RS by explicitely registering them

Switching on/off automatic discovery of resource classes and providers in JAX-RS

The automatic discovery may complicate things when provider classes are included in the libraries used by an application or preinstalled in a server, for example Jackson-related jar in Wildfly. So I prefer to switch off every features I am not aware of. The JAX-RS specification states:

  • When an Application subclass is present in the archive, if both Application.getClasses and Application.getSingletons return an empty collection then all root resource classes and providers packaged in the web application MUST be included and the JAX-RS implementation is REQUIRED to discover them automatically by scanning a .war file as described above.
  • If either getClasses or getSingletons returns a non-empty collection then only those classes or singletons returned MUST be included in the published JAX-RS application.

So, essentially if methods getClasses and getSingletons are not overriden the resource classes and providers are discovered automatically. Let's use two root resource classes to illustrate the rule. The full illustration code is available in Github.

@Path("/registered")
public class MyRegisteredResource {

    @GET
    public String getBook() {
        return "Hello Registered World!";
    }
}

@Path("/unregistered")
public class MyUnregisteredResource {

    @GET
    public String getBook() {
        return "Hello Unregistered World!";
    }
}

Both resources operate if the Application class is empty:

@ApplicationPath("/api")
public class MyApplication extends Application {
}

If I override getClasses method, only the resource class returned by the method will function:

@ApplicationPath("/api")
public class MyApplication extends Application {

    @Override
    public Set<Class<?>> getClasses() {
        return new HashSet<Class<?>>() {
            {
                add(MyRegisteredResource.class);
            }
        };
    }
}
In which method to register a class?

Quotes from the JAX-RS specification on the Lifecyle of providers and resource classes :

  • By default a new resource class instance is created for each request to that resource.
  • By default a single instance of each provider class is instantiated for each JAX-RS application.
  • By default, just like all the other providers, a single instance of each filter or entity interceptor is instantiated for each JAX-RS application.

So the root resource classes should be returned by getClasses, whereas providers, including filters, by getSingletons method.

Getting the standard servlet defined types such as HttpServletRequest in ContainerRequestFilter

The worst feature of JAX-RS filters is there is not straightforward way to access HttpServletRequest instance. The reference to HttpServletRequest can be injected into managed classes using @Context annotation. However, according to the specification the filters have to be instantiated singletons. That means that injection will not work.

If you want to access in a filter any of the standard servlet arguments such as HttpServletRequest, HttpServletResponse, ServletConfig, ServletContext, the filter will have to be registered in getClasses, so that its instance is created and injected for each request. Otherwise injection is impossible and without it there is no way to access the servlet-defined types.

Monday, January 29, 2018

Aligning horizontally and vertically a div with absolute position and unknown size inside a div

When the size of the div with absolute position is unknown, the simplest solution is using translate function. Obviously, the container is not static.

.absolute1 {
    position:absolute;
    background-color: antiquewhite;     
    top: 50%;
    left: 50%;
    transform: translate(-50%, -50%);
}

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Element with absolute position and unknown size

Element with absolute position and unknown size

Element with absolute position and unknown size

By the way, many authors suggest a solution that works only for divs with height and width set:

.absolute2{
    position:absolute;
    background-color: antiquewhite;    
    top: 0; 
    left: 0; 
    bottom: 0; 
    right: 0;
    margin: auto;
    width:50%;
    height:30%;
}

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Relative Container, Relative Container, Relative Container, Relative Container, Relative Container

Element with absolute position and known size

Element with absolute position and known size

Element with absolute position and known size

Saturday, January 27, 2018

Enabling SSL in Wildfly using a free certificate from Let's Encrypt

Let’s Encrypt is a free Certificate Authority. To enable HTTPS on a website, one needs to get a certificate from a Certificate Authority. Let’s Encrypt recommends to use Certbot - a tool that validates your ownership over the target domain and fetches the certificates.

Intalling Certbot on CentOS7
sudo yum install epel-release
sudo yum install certbot
Validating a domain to fetch a certificate

Fetching certificates is described in detail in thethe official documentation, which needs to be consulted to understand the command below. Briefly, Certbot attempts to validate that you own the domain for which you ask a certificate. When a user opts for http-based verification, Certbot asks to create two files with tiny contents specified by Certbot so that they are accessible at some urls in your domain also specified by Certbot.

sudo certbot certonly --manual --preferred-challenges http -d food-diary-online.com -d www.food-diary-online.com --manual-auth-hook /opt/SSLCertificates/authenticator.sh --non-interactive --manual-public-ip-logging-ok

--manual-auth-hook option points on the script authenticator.sh creating files in a web application folder .well-known/acme-challenge that is accessible by CertBot via web:

#!/bin/bash

TARGET_DIR=/opt/wildfly-10.1.0.Final/standalone/deployments/FoodApp.war/.well-known/acme-challenge
mkdir -p $TARGET_DIR
echo $CERTBOT_VALIDATION > $TARGET_DIR/$CERTBOT_TOKEN

As the result the certificates are downloaded

Strangely, a normal user could not access the certificates because of the permissions on the created by Certbot parent folders. So I adjusted the permissions:

sudo chmod 755 /etc/letsencrypt/archive
sudo chmod 755 /etc/letsencrypt/live
cat /etc/letsencrypt/live/food-diary-online.com/fullchain.pem

The indicated directory /etc/letsencrypt/live/food-diary-online.com/ contains symbolic links privkey.pem and fullchain.pem to the most recently downloaded certificate files. For example, when I downloaded the certificates second time, the directory contents were:

What happens without manual-auth-hook

Certbot asks to create two files so that they are accessible at the specified urls. Without the authenticator script, the output is like:

Create a file containing just this data:

9-gBwqnje4DmxbrxaXX7E3-Rua2_-rY54JB6wsdCWqo.m1NBHzDLwknVhXjqDceEqOyC2Na8q0e1QJws4FCqErs

And make it available on your web server at this URL:

http://food-diary-online.com/.well-known/acme-challenge/9-gBwqnje4DmxbrxaXX7E3-Rua2_-rY54JB6wsdCWqo

--Press Enter to Continue--

Create a file containing just this data:

VNDE0iHhEQccJuFcDFF3X-FwsaxItyFlfE0GGy_6ixI.m1NBHzDLwknVhXjqDceEqOyC2Na8q0e1QJws4FCqErs

And make it available on your web server at this URL:

http://www.food-diary-online.com/.well-known/acme-challenge/VNDE0iHhEQccJuFcDFF3X-FwsaxItyFlfE0GGy_6ixI

--Press Enter to Continue--


IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/food-diary-online.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/food-diary-online.com/privkey.pem
   Your cert will expire on 2018-04-26. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"

During this process I just created the requested files in a war deployed into the Wildfly:

Importing the private key with the fetches certificate into a Java key store (jks)

The following files has been created:

  • privkey.pem - Private key for the certificate.
  • fullchain.pem - The server certificate followed by intermediate certificates that web browsers use to validate the server certificate.

However, Wildfly 10 accepts only jks. So the fullchain.pem has to be imported into jks. However, keytool can import a certificate or an entire keystore, but does not import a private key separated from the paired public key with the certificate. Therefore, a private key has to be combined with the certificate in a acceptable PKCS12 keystore with openssl command. Then the keystore can be imported into jks. The keystore will be created in /opt/SSLCertificates/

cd /opt/SSLCertificates/
openssl pkcs12 -export -in /etc/letsencrypt/live/food-diary-online.com/fullchain.pem -inkey /etc/letsencrypt/live/food-diary-online.com/privkey.pem -out keystore.p12 -name wildfly -passout pass:changeit

changeit is the password for the keystrore to be created. File keystore.p12 is created.

keytool -importkeystore -deststorepass changeit -destkeypass changeit -destkeystore keystore.jks -srckeystore keystore.p12 -srcstoretype PKCS12 -srcstorepass changeit -v -noprompt

Jks keystore.jks is created when the command is executed for the first time. During the second import the existing alias will be overwritten:

Configuring SSL in Wildfly

Stop the server and edit standalone.xml so that it contains:

<security-realm name="ApplicationRealm">
    <server-identities>
     <ssl>
            <keystore path="/opt/SSLCertificates/keystore.jks" keystore-password="changeit" alias="wildfly" key-password="changeit"/>
        </ssl>
    </server-identities>
    <authentication>
        <local default-user="$local" allowed-users="*" skip-group-loading="true"/>
        <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
    </authentication>
    <authorization>
        <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
    </authorization>
</security-realm>

Start the server. Well, that's it, we're done.

Automatic certificate renewal Incomplete yet

The only problem with the Let's Encrypt certificates is that they last for 90 days, so they have to be regularly renewed. This can be achieved with a script scheduled in crontab.

cerbot renew command attempts to renew any previously-obtained certificates that expire in less than 30 days. The same plugin and options that were used at the time the certificate was originally issued will be used for the renewal attempt, unless you specify other plugins or options. renew can be run as frequently as you want since it will usually take no action.

I created deployhook.sh script in /opt/SSLCertificates/. The script merely executes the commands used above to import the certificates into the java keystore and restart wildfly.

#!/bin/bash

service wildfly stop

cd /opt/SSLCertificates/
openssl pkcs12 -export -in /etc/letsencrypt/live/food-diary-online.com/fullchain.pem -inkey /etc/letsencrypt/live/food-diary-online.com/privkey.pem -out keystore.p12 -name wildfly -passout pass:changeit

keytool -importkeystore -deststorepass changeit -destkeypass changeit -destkeystore keystore.jks -srckeystore keystore.p12 -srcstoretype PKCS12 -srcstorepass changeit -v -noprompt

service wildfly start

Now, to renew certificates one single-line command is sufficient. The script indicated by --deploy-hook is executed only after a successful certificate renewal.

sudo certbot renew --deploy-hook deployhook.sh

The output of the command:

[centos@ip-172-31-42-159 SSLCertificates]$ sudo certbot renew --deploy-hook ./deployhook.sh
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/food-diary-online.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator manual, Installer None
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for food-diary-online.com
http-01 challenge for www.food-diary-online.com
Waiting for verification...
Cleaning up challenges
Running deploy-hook command: ./deployhook.sh
Output from deployhook.sh:
Stopping wildfly (via systemctl):  [  OK  ]
Starting wildfly (via systemctl):  [  OK  ]

Error output from deployhook.sh:
Warning: Overwriting existing alias wildfly in destination keystore
Entry for alias wildfly successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
[Storing keystore.jks]


-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/food-diary-online.com/fullchain.pem
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/food-diary-online.com/fullchain.pem (success)
-------------------------------------------------------------------------------
[centos@ip-172-31-42-159 SSLCertificates]$

To automate the renewals, the command has to be scheduled in the crontab.

Thursday, January 25, 2018

Login into a web application with Facebook via redirect

Like Google, the Facebook documentation on login includes two options: using Facebook Javascript SDK and without it. I used the second option, manually building a login flow, to create a Javascript-free login flow for the back-end.

In my simple web application the flow is as follows:

  • When the login button is click, the user is redirected to the facebook page. The login button is inside an anchor tag:

    <a href="https://www.facebook.com/v2.12/dialog/oauth?client_id=407938939637716&redirect_uri=http://localhost:8080/test/facebook&scope=email&response_type=code&state=http://localhost:8080/test/|1462070876
    ">

    The url contains the following parameters:

    • client_id from the app's dashboard.
    • redirect_uri will receive the response from the Facebook login. The uri must be whitelisted in the App Dashboard.
    • state maintains state between the request and callback. It will be appended unchanged to the redirect_uri.
    • response_type:
      • code - the response data is included as URL parameters and contains code parameter. The data is accessible to the back-end.
      • token - the response data is included as a URL fragment and contains an access token. The response hash can be accesses only on the client by javascript.
    • scope - a list of Permissions to request from the person using your app. Note, even if you request the email permission it is not guaranteed you will get an email address. For example, if someone signed up for Facebook with a phone number instead of an email address, the email field may be empty.
  • After a login attempt, the browser is redirected to the redirect_uri with appended response paremeters code and state:

    http://localhost:8080/test/facebook?code=AQAlQPnW9Vbft-F8DW_ybmgjnk5fJ9ok7YiOglNDvougkpDUYyOC4D5gdhWQ4o54cYcVE9bihhvKjO6HRIQCXNIhLW6jUlaCvehwScSXbE9U3zDKBsbJo-uvMgPc9xJbzKAunmIdr3dDjJ72-SKqipvEYTkHtKksbVDEfbUR4DRL6ei8SQyf7A-8ULAGhZhJAgLsqfKYkqal2GzgbxtK3npvBS1OiWZFZvlGirHPbnOpM80EO5E7WDBqG7GsSR9c6lM6Xudehpo7U9OacW5h2XDwIuVTCFg3pNgtiptkotEimhBgigdgWTLFJJgYzFhYRrfj3O-ksbcAknd_nqUvYFfgewJo-5ejJ4HL1fGdzlrF-A&state=http%3A%2F%2Flocalhost%3A8080%2Ftest%2F%7C1462070876

    The state values in the original request and the response are the same. The code has to be included in a GET request to another endpoint. Additionally, client_id, redirect_uri used in the initial request, and client_secret from the App Dashboard are required.

    https://graph.facebook.com/v2.12/oauth/access_token?code=AQAlQPnW9Vbft-F8DW_ybmgjnk5fJ9ok7YiOglNDvougkpDUYyOC4D5gdhWQ4o54cYcVE9bihhvKjO6HRIQCXNIhLW6jUlaCvehwScSXbE9U3zDKBsbJo-uvMgPc9xJbzKAunmIdr3dDjJ72-SKqipvEYTkHtKksbVDEfbUR4DRL6ei8SQyf7A-8ULAGhZhJAgLsqfKYkqal2GzgbxtK3npvBS1OiWZFZvlGirHPbnOpM80EO5E7WDBqG7GsSR9c6lM6Xudehpo7U9OacW5h2XDwIuVTCFg3pNgtiptkotEimhBgigdgWTLFJJgYzFhYRrfj3O-ksbcAknd_nqUvYFfgewJo-5ejJ4HL1fGdzlrF-A&client_id=407938939637716&client_secret=7fd6fa037e097fcf002fb350b815f5b3&redirect_uri=http://localhost:8080/test/facebook

    The response is a Json containing an access_token.

    {"access_token":"EAAFzBKZBWT9QBABhRNBOKwQZBMza2siRzUZBeQHi0Jp2qrXijTEnbVijy4ZAjnTzBzigZA11Vh4RhZBZB2DDu1n8fEJBdBh5mxFETqJJFLLHQnkG0XyffthKg2tU62csiCvsCHxiVOMuZAWy2lS74sKLTbL8sC8EgLEZD","token_type":"bearer","expires_in":5181583}

    That the difference from Google sign in. In Google additionally id_token with the user's detail is received. In Facebook an additional request is required.

  • Using the received access token, the sever-side code makes yet another GET request retrieves a user email from the Graph API. The /me node is a special endpoint that translates to the user_id of the person whose access token is currently being used to make the API calls. Access tokens are portable. Graph API calls can be made from clients or from your server on behalf of clients. The calls to the Graph API are better secured by adding appsecret_proof parameter.

    https://graph.facebook.com/v2.12/me?access_token=EAAFzBKZBWT9QBABhRNBOKwQZBMza2siRzUZBeQHi0Jp2qrXijTEnbVijy4ZAjnTzBzigZA11Vh4RhZBZB2DDu1n8fEJBdBh5mxFETqJJFLLHQnkG0XyffthKg2tU62csiCvsCHxiVOMuZAWy2lS74sKLTbL8sC8EgLEZD&debug=all&fields=email&format=json&method=get&pretty=0&appsecret_proof=34b9499185c81a1f37a68cb3b012ae2dcb6882d8066dc58c88641247c28bbce9
    

    The response is a json:

    {"email":"dummy000@mail.ru","id":"10215579219697013","__debug__":{}}

    The retrieved email is used to log the user into the web application.

In App Dashboard I used such security settings:

The option below allows only the API calls that either include appsecret_proof or are made from the same device the token was issued.

Wednesday, January 24, 2018

Generating MD5, SHA-1, SHA-256, SHA-384, SHA-512 message digests

Just a note within sight. To generate hashes using any possible algorithms, I use digest method of a Java class:

public class MessageHash {

    static String DEFAULT_ALGORITHM = "SHA-1"; // MD5, SHA-1, SHA-256, SHA-384, SHA-512

    static String digest(String input, String algorithm) throws NoSuchAlgorithmException {
        MessageDigest md = MessageDigest.getInstance(algorithm);
        return HexConverter.bytesToHex(md.digest(input.getBytes()));
    }

    public static String digest(String input) {
        try {
            return digest(input, DEFAULT_ALGORITHM);
        } catch (NoSuchAlgorithmException ex) {
            throw new RuntimeException("This is impossible");
        }
    }
}

HmacSHA256 - sha256 hash using a key (for appsecret_proof in Facebook)

Unlike Google, in Facebook access tokens are portable - they can be used without client or app id. To kind of protect or rather label them, all Graph API calls from a server (only server) should secured by adding a parameter appsecret_proof set to the sha256 hash of the access token generated using the app secret as the key.

Here is an example of how to do it on a Java server:

public class Sha256Digest {

    Mac mac;
     
    Sha256Digest() throws UnsupportedEncodingException, NoSuchAlgorithmException, InvalidKeyException {
        this(APP_SECRET);
    }

    Sha256Digest(String key) throws UnsupportedEncodingException, NoSuchAlgorithmException, InvalidKeyException {
        SecretKeySpec sk = new SecretKeySpec(key.getBytes(StandardCharsets.UTF_8.toString()), "HmacSHA256");
        mac = Mac.getInstance("HmacSHA256");
        mac.init(sk);
    }

    String hash(String msg) throws UnsupportedEncodingException {
        return HexConverter.bytesToHex(mac.doFinal(msg.getBytes(StandardCharsets.UTF_8.toString())));
    }

    public static void main(String[] args) throws Exception {
        System.out.println(new Sha256Digest().hash("Test"));
    }
}

For converting an array of bytes into a string of hexadecimal values I use an additional class:

public class HexConverter {

    private final static char[] HEXARRAY = "0123456789abcdef".toCharArray();

    public static String bytesToHex(byte[] bytes) {
        char[] hexChars = new char[bytes.length * 2];
        for (int j = 0; j < bytes.length; j++) {
            int v = bytes[j] & 0xFF;
            hexChars[j * 2] = HEXARRAY[v >>> 4];
            hexChars[j * 2 + 1] = HEXARRAY[v & 0x0F];
        }
        return new String(hexChars);
    }
}

The resulting string can be included in Facebook calls like:

https://graph.facebook.com/v2.11/me?access_token=EAAFzBKZBWT9QBAFXEDBfdKu8Q7cDXZAWXSaZAIKuDZB04A5mAlCTMpXKgBJNd42MXZAo5Gk8ZAv8u6mjCXGLfLTjT6ORikMLWOCFTxbaxHqcOfpJU7iGIyk5xKozSv0HG4ctm0wpE2xHriZCeITEQZAKWbHoveuj2xbGSBvdPhE8uX5HXtEdgUkc82XNZAuQSLi8ZD&debug=all&fields=email&format=json&method=get&pretty=0&appsecret_proof=734e6e019eb20821682797320845f1df2e813f01cc779cbcd94bb55a9a37457f

To prevent API calls lacking the proof, Require App Secret switch should be activated in the application settings. Only allow calls from a server and require app secret or app secret proof for all API calls.

Using JSON-P to parse heterogeneous JSON in HTTP responses

Suppose you need to query Facebook Graph API. The responses to your HTTP requests have JSON format. A very convinient Java API for JSON Processing helps to parse and query the heterogeneous JSON responses.

For example, I try to get an email of the user whose access token was obtained after the user's login into my application. For this, I access url like:

https://graph.facebook.com/v2.11/me?access_token=EAAFzBKZBWT9QBAHzWcGGSy5GepjlS9S1YEPvN1p2jwaGxc0QZCaVoAZCmsZB8YaE1AkegbmObdBDY64DDD1t1kxezOgpEFKbbLKlyQyPcEiyUZCwSI3iJOhe9ioahZA9Ye6hvOybhzGeOODFdihEnPbuw5sso5CzPEZAQL1RkdM3cfKajOdKsPmMWOvNhrDtE0ZD&debug=all&fields=email&format=json&method=get&pretty=0

The reponse is a JSON with some hexadecimal digits encoding @ character:

{"email":"marian.caikovski\u0040mail.ru","id":"10215579219697013","__debug__":{}}

To easily execute an HTTP request, parse the response and get the decoded email property I use:

String readUserEmailFromGraphAPI(String token) throws IOException {
    try (JsonReader jsonReader = Json.createReader(
            new InputStreamReader(
                    new URL("https://graph.facebook.com/v2.11/me?access_token=" + token + "&debug=all&fields=email&format=json&method=get&pretty=0")
                            .openStream()))) {
        JsonObject obj = jsonReader.readObject();
        return obj.getString("email");
    }
}

How to get a request url hash on the back end server. Reconstructing the full request url in a servlet.

It is impossible, the browser does not include the hash into the request url sent to the server:

Just a note for myself on what values of the request path can be extracted from HttpServletRequest in a servlet:

The full request path can be reconstructed by a function like:

request.getRequestURL() + (request.getQueryString() != null ? ("?" + request.getQueryString()) : "")

Monday, January 22, 2018

How to activate gzip compression of selected content types in Tomcat or Wildfly

Another note for myself. To enable gzip compression in tomcat add an additional attribute to Connector tag in CATALINA_HOME/conf/server.xml:

<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="443" compressibleMimeType="application/javascript,text/css,application/json" compression="on"/>

Wildfly is not so well documented as Tomcat. So this note assembled from pieces of information saves time. Essentially one needs to enable and configure gzipFilter using Undertow predicates. Edit the default configuration file standalone.xml:

<subsystem xmlns="urn:jboss:domain:undertow:3.1">
    <buffer-cache name="default"/>
    <server name="default-server">
        <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>
        <https-listener name="https" socket-binding="https" security-realm="ApplicationRealm" enable-http2="true"/>
        <host name="default-host" alias="localhost">
            <location name="/" handler="welcome-content"/>
            <access-log pattern="%h %t "%r" %s "%{i,User-Agent}"" prefix="myaccess."/>
            <filter-ref name="gzipfilter" predicate="regex[pattern='text/html|text/css|application/javascript|application/json',value=%{o,Content-Type}] and max-content-size[value=1024]"/>
        </host>
    </server>
    <servlet-container name="default">
        <jsp-config/>
        <persistent-sessions path="sessions" relative-to="jboss.server.temp.dir"/>
        <websockets/>
    </servlet-container>
    <handlers>
        <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
    </handlers>
    <filters>
        <gzip name="gzipfilter"/>
    </filters>
</subsystem>

All the possible predicates are listed in Undertow documentation. Some people use url-based predicates like:

<filter-ref name="gzipFilter" predicate="path-suffix['.css'] or path-suffix['.js']" />

Alternatively, one can use a custom gzip compression servlet filter that can be more easily configured to target some specific output. A working example is in GitHub. I keed this sample only because it works well and its GIPOutputStream potentially could be replaced by some other stream to for example encrypt the output or produce hashes.

Google Sign in into a website using redirect ux_mode

Google Javascript client library used for sign in is built on the OpenID Connect protocol, which is straightforward. The library uses the implicit flow whereby tokens are passed in url hash. It is not a good option for server side authentication. It differs from a less complicated basic/server flow in which tokens are passed as url parameters. The server flow I describe in a separate post.

Google Sign-In for Websites documentation provides only examples where users sign in via a Google popup. I adapted their code so that another redirect, which is another consent flow option, is used. I also added a primitive backend code that process the ID token. In my sample web application saved to GitHub, the entire consent flow happens in one window without any popups because the initialization is launched with following parameters:

gapi.auth2.init({
            client_id: clientId,
            fetch_basic_profile: false, 
            scope: 'email',
            ux_mode: 'redirect', 
            redirect_uri: 'http://localhost:8080/test/' 
        })

The application can be deployed to Tomcat or anywhere, but first a client id should be generated in google API console and copied to Constants class.

For the unauthenticated users the welcome page displays only the standard Google Sign-In button that meets the strict Google branding guidelines.

On clicking the button the browser is redirected to Google authentication page.

If the user has only one account in Google and he is already signed in, he is immediatly redirected by to the original page. Otherwise, the user has to select with what account to sign in and then upon authentication, the user is redirected back to the original page. To imitate a complete process of authentication, the page forwards the received from google ID token to the REST resource in the Java backend. The backend process the id, and sends back a JSON with the user's email. So for the authenticated users the only page displays their email received from the Java backend and a link for signing out.

Thursday, January 18, 2018

Resizing selected pictures a browser before uploading them to a REST resource in a backend

The sample application uploads multipart data comprising data from several text inputs together with several photo files select in file type input to a JAX-RS rest resource. Note, multipart data is not mentioned in JAX-RS specification. So the back end uses RESTEasy-specific features. Before uploading, the files are resized in the browser. Then, they are scaled down to thumbnails in the back end.

The web application is adapted for Wildfly, but it works as well with Tomcat if the scope of RESTEasy-related dependencies is changed from provided to the default by removing it.

How to style a file input

The input will accept multiple but only images. One cannot change much the file type input. A workaround is to use a label tag and hide the input with css. Note, the ugly styling here serves merely to demonstrate that styling is possible.

<label id='dropbox' for='fileInput'><img src="imgs/File-Upload-icon.png"/>Select photos</label>
<input id='fileInput' type="file" accept="image/*" multiple />

A sample css:

input[type=file] {
    display: none;
}
label img {
    max-height: 1.5em;
}

label {
    border: 1px solid;
    display: inline-block;
    padding: 0.3em;
}
Resizing selected files using canvas and its function onBlob

The unique resized files are stored in an array:

var selectedFiles = []; // the array with the unique resized files that will be uploaded

When new pictures are selected using the file input, a change event listener is invoked:

$('input[type=file]').change(function () {
    resizeAndShowThumbs(this.files);
});
function resizeAndShowThumbs(files) {
    for (var c = 0; c < files.length; c++) {
        var file = files[c];
        if (file.type.startsWith("image/") && isFileNotYetIncluded(file)) {
            resize(file, showThumb);
        }
    }
}
function isFileNotYetIncluded(file) {
    for (var c = 0; c < selectedFiles.length; c++) {
        if (selectedFiles[c].originalNameSize.equals(file)) { // file has name and size read-only properties
            return false;
        }
    }
    return true;
}

The event listener calls the resize function only if a file is not yet included in the array. The files are identified by their names and initial sizes. After a file is resized the callback showThumb is called.

function showThumb(file) {
    selectedFiles.push(file);
    showMessage();
    $previewList.append('<li><p>' + file.originalNameSize.name + '</p><img src="' + URL.createObjectURL(file)
            + '"  onload="window.URL.revokeObjectURL(this.src);"/></li>');
}

The resized picture have jpeg compression. The problem with resizing is that sometimes a resized jpeg-compressed file is has a bigger size than the source file with bigger dimensions. So the file with smaller size is selected between the source and resized files. On the back-end the pictures are converted into thumbnails using ImageIO class, which accepts only jpg, bmp, gif, png formats. In the unlikely case of the source file having an unacceptable format, the resized jpeg file will be uploaded even if it is bigger.

var MAX_SIZE = 1200, MIME = 'image/jpeg', JPEG_QUALITY = 0.95;
// the files types accepted by java ImageIO
var acceptableTypes = ["image/gif", "image/png", "image/jpeg", "image/bmp"]; 

function size(size) {
    var i = Math.floor(Math.log(size) / Math.log(1024));
    return (size / Math.pow(1024, i)).toFixed(2) * 1 + ['b', 'kb', 'Mb'][i];
}

function resizePhoto(file, callback) {
    var image = new Image();
    image.onload = function ( ) {
        URL.revokeObjectURL(this.src);
        var canvas = document.createElement('canvas');
        var width = this.width;
        var height = this.height;

        if (width > height) {
            if (width > MAX_SIZE) {
                height *= MAX_SIZE / width;
                width = MAX_SIZE;
            }
        } else {
            if (height > MAX_SIZE) {
                width *= MAX_SIZE / height;
                height = MAX_SIZE;
            }
        }

        canvas.width = width;
        canvas.height = height;
        canvas.getContext('2d').drawImage(image, 0, 0, width, height);
        canvas.toBlob(callback.bind(null, this.width, this.height, width, height), MIME, JPEG_QUALITY);
    };
    image.src = URL.createObjectURL(file);
}


function chooseSmallerFile(file, resizedFile) {
    if (file.size > resizedFile.size) {
        console.log('the resized file is smaller');
        return resizedFile;
    } else {
        // resized is bigger than the original
        // however, java ImageIO supports only  jpg, bmp, gif, png, which perferctly match mime types, the front-end should send only those types
        // if the file type is none of image/gif, image/png, image/jpeg, image/bmp use the bigger resized file
        console.warn('resized is bigger the the original');
        if (acceptableTypes.indexOf(file.type) >= 0) {
            return file;
        } else {
            console.warn('but the source file type is unacceptable: ' + file.type);
            return  resizedFile;
        }
    }
}

 function resize(file, callback) {
    resizePhoto(file, function (originalWidth, originalHeight, resizedWidth, resizedHeight, resizedFile) {
        console.log('filename=' + file.name + '; size=' + size(file.size) + '=>' + size(resizedFile.size)
                + '; dimensions=' + originalWidth + '/' + originalHeight + '=>' + resizedWidth + '/' + resizedHeight);
        var smallerFile = chooseSmallerFile(file, resizedFile);
        smallerFile.originalNameSize = new NameAndSize(file.name, file.size); // name is erased in the resized file. the name and size are used to select unique files
        callback(smallerFile);
    });
};

The resizing code produces in the console lots of file size related debug messages. For example, when many pictures coming from different sources are selected:

The console messages indicate that sometimes it is cheaper to upload the original file with the bigger dimensions:

Dragging and dropping photos

Instead of clicking the file input label, one can drop on it the files dragged from any file browser. To implement drag and drop, only few lines are required:

$('#dropbox').on("dragenter", onDragEnter).on("dragover", onDragOver).on("drop", onDrop);

function onDragEnter(e) {
    e.stopPropagation();
    e.preventDefault();
}

function onDragOver(e) {
    e.stopPropagation();
    e.preventDefault();
}

function onDrop(e) {
    e.stopPropagation();
    e.preventDefault();
    resizeAndShowThumbs(e.originalEvent.dataTransfer.files);
}

How the resized photos together with values from other inputs can be posted as multipart form data to a REST resource is described in a separate post, because this one would be to long.